Monitoring system and monitoring method

ABSTRACT

In a pilotless flying object detection system, a masking area setter sets a masking area to be excluded from detection of a pilotless flying object which appears in a captured image of a monitoring area, based on audio collected by a microphone array. An object detector detects the pilotless flying object based on the audio collected by the microphone array and the masking area set by the masking area setter. An output controller superimpose sound source visual information, which indicates the volume of a sound at a sound source position, at the sound source position of the pilotless flying object in the captured image and displays the result on a first monitor in a case where the pilotless flying object is detected in an area other than the masking area.

BACKGROUND

1. Technical Field

The present disclosure relates to a monitoring system and a monitoringmethod which monitor an imaging area of a camera device in which, forexample, a pilotless flying object flies.

2. Description of the Related Art

A flying object monitoring apparatus depicted in Japanese PatentUnexamined Publication No. 2006-168421 is capable of detecting thepresence of an object and the flight direction of the object using aplurality of audio detectors which detect sounds generated in amonitoring area on a per-direction basis. If a processor of the flyingobject monitoring apparatus detects the flight and the flight directionof a flying object through audio detection using microphones, theprocessor causes a monitoring camera to face the direction in which theflying object flies. Furthermore, the processor displays a video whichis captured by the monitoring camera on a display device.

However, when a pilotless flying object such as a drone is detected, ifaudio detection is performed by setting all directions in the monitoringarea as a monitoring target, from the perspective of a non-directionalmicrophone, a direction in which a frequency of a loud sound beinggenerated is high cannot be set as a masking area. Therefore, in a casewhere a loud sound is detected in the monitoring area, an object in adirection of the masking area, which is different from a pilotlessflying object originally desired to be detected, may be erroneouslydetected as a target pilotless flying object. In addition, if a maskingarea that is excluded from being a target is set in advance instead ofsetting all directions in the monitoring area as a monitoring target, itcan be expected that detection of the pilotless flying object originallydesired to be detected will be performed faster.

In a monitoring camera which changes an imaging direction in order toperform imaging by focusing on the detected flying object, it isdifficult to visually present, to a user, the location in the imagingarea of the camera device where the pilotless flying object is detected,and what kinds of sound source are present at which locations in thesame imaging area.

In addition, in a case where a sound pressure in a frequency unique tothe flying object such as a helicopter or a Cessna is greater than orequal to a predetermined set level, if the flying object is determinedto be the monitoring target, when any sound is detected in the imagingarea of the camera device, it is difficult to specifically present thevolume of the sound as detailed visual information for sound, regardlessof the magnitude of the volume of the detected sound at a sound sourceposition.

SUMMARY

The disclosure aims to suppress deterioration of the detection accuracyof a pilotless flying object and to improve a detection process of apilotless flying object by setting a masking area to be excluded from adetection process of a pilotless flying object as a detection target, inan imaging area of a camera device.

The disclosure aims to visually present, to a user, the location in theimaging area of the camera device where the pilotless flying object isdetected, and what kinds of sound source are present at which locationsin the same imaging area without deterioration of the visibility of thecaptured image of the camera device.

The disclosure aims to present in detail, in stages, the volume of thedetected sound at the sound source position in the imaging area of thecamera device, regardless of the magnitude of the volume of the sound atthe sound source position, and to assist the user in accuratelyascertaining the volume of the sound at the sound source position.

According to the disclosure, there is provided a monitoring systemincluding a camera which images an imaging area; a microphone arraywhich collects audio of the imaging area; a monitor which displays acaptured image of the imaging area which is captured by the camera; amasking area setter that sets a masking area to be excluded fromdetection of a pilotless flying object which appears in the capturedimage of the imaging area, based on the audio collected by themicrophone array; a detector that detects the pilotless flying objectbased on the audio collected by the microphone array and the maskingarea set by the masking area setter; and a signal processor thatsuperimposes a sound source visual image, which indicates the volume ofa sound at a sound source position, at the sound source position of thepilotless flying object in the captured image and displays the result onthe monitor in a case where the pilotless flying object is detected inan area other than the masking area.

According to the disclosure, there is provided a monitoring method in amonitoring system provided with a camera and a microphone array, themethod including: imaging an imaging area by the camera; collectingaudio of the imaging area by the microphone array; displaying a capturedimage of the imaging area which is captured by the camera, on a monitor;setting a masking area to be excluded from detection of a pilotlessflying object which appears in the captured image of the imaging area,based on the audio collected by the microphone array; detecting thepilotless flying object based on the audio collected by the microphonearray and the set masking area; and superimposing a sound source visualimage, which indicates the volume of a sound at a sound source position,at the sound source position of the pilotless flying object in thecaptured image and displaying the result on the monitor in a case wherethe pilotless flying object is detected in an area other than themasking area.

According to the disclosure, since the masking area to be excluded fromthe detection process of the pilotless flying object as the detectiontarget can be set in the imaging area of the camera device, it ispossible to suppress deterioration of the detection accuracy of thepilotless flying object and to improve the detection process of thepilotless flying object.

According to the disclosure, there is provided a monitoring method in amonitoring system provided with a camera and a microphone array, themethod including: imaging an imaging area by the camera; collectingaudio of the imaging area by the microphone array; displaying a capturedimage of the imaging area which is captured by the camera, on a monitor;deriving a sound parameter, which specifies the volume of a sound of theimaging area, on a per-predetermined-unit basis of pixels which form thecaptured image of the imaging area, based on the audio collected by themicrophone array; generating a sound parameter map as a transparent mapin which a sound source visual image, in which the sound parameter isconverted into a visual image according to comparison between thederived sound parameter and a threshold relating to the volume of asound, on a per-predetermined-unit basis of pixels, is linked tocorrespond to the size of the captured image of the imaging area; andsuperimposing the generated translucent map onto the captured image ofthe imaging area and displaying the result on the monitor.

According to the disclosure, there is provided a monitoring systemincluding a camera which images an imaging area; a microphone arraywhich collects audio of the imaging area; a monitor which displays acaptured image of the imaging area which is captured by the camera; asound parameter deriving unit that derives a sound parameter, whichspecifies the volume of a sound of the imaging area, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area, based on the audio collected by the microphone array;and a signal processor that generates a sound parameter map as atranslucent map in which a sound source visual image, in which the soundparameter is converted into a visual image according to comparisonbetween the sound parameter derived by the sound parameter deriving unitand a threshold relating to the volume of a sound, on aper-predetermined-unit basis of pixels, is linked to correspond to thesize of the captured image of the imaging area, in which the signalprocessor superimposes the translucent map onto the captured image ofthe imaging area and displays the result on the monitor.

According to the disclosure, it is possible to present in detail, instages, the volume of the detected sound at the sound source position inthe imaging area of the camera device, regardless of the magnitude ofthe volume of the sound at the sound source position, and to assist theuser in accurately ascertaining the volume of the sound at the soundsource position.

According to the disclosure, there is provided a monitoring systemincluding a camera which images an imaging area; a microphone arraywhich collects audio of the imaging area; a monitor which displays acaptured image of the imaging area which is captured by the camera; asound parameter deriving unit that derives a sound parameter, whichspecifies the volume of a sound of the imaging area, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area, based on the audio collected by the microphone array;a threshold adjuster that changes a setting of a correspondencerelationship between each threshold of a plurality of thresholdsdefining the volume of a sound in stages and a sound source visual imagein which the sound parameter is converted in stages into a differentvisual image according to comparison between the sound parameter andeach threshold, according to the captured image of the imaging area; anda signal processor that superimposes the sound source visual imagecorresponding to the sound parameter onto the captured image of theimaging area, on a per-predetermined-unit basis of pixels which form thecaptured image of the imaging area, based on the sound parameter derivedby the sound parameter deriving unit and the correspondence relationshipchanged by the threshold adjuster and displays the result on themonitor.

According to the disclosure, there is provided a monitoring systemincluding: a camera which images an imaging area; a microphone arraywhich collects audio of the imaging area; a monitor which displays acaptured image of the imaging area which is captured by the camera; asound parameter deriving unit that derives a sound parameter, whichspecifies the volume of a sound of the imaging area, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area, based on the audio collected by the microphone array;and a signal processor that superimposes a sound source visual image inwhich the sound parameter is converted in stages into a different visualimage according to comparison between the sound parameter derived by thesound parameter deriving unit and a plurality of thresholds relating tothe volume of a sound, on a per-predetermined-unit basis of pixels whichform the captured image of the imaging area and displays the result onthe monitor, in which, when any sound source position is designated inthe captured image of the imaging area on which the sound source visualinformation is superimposed, the sound parameter deriving unit derivesthe sound parameter for each value obtained by dividing a predeterminedunit of pixels which form a rectangular range including the sound sourceposition by a ratio between sizes of the captured image of the imagingarea and the rectangular range.

According to the disclosure, there is provided a monitoring method in amonitoring system provided with a camera and a microphone array, themethod including: imaging an imaging area by the camera; collectingaudio of the imaging area by the microphone array; displaying a capturedimage of the imaging area which is captured by the camera, on a monitor;deriving a sound parameter, which specifies the volume of a sound of theimaging area, on a per-predetermined-unit basis of pixels which form thecaptured image of the imaging area, based on the audio collected by themicrophone array; changing a setting of a correspondence relationshipbetween each threshold of a plurality of thresholds defining the volumeof a sound in stages and a sound source visual image in which the soundparameter is converted in stages into a different visual image accordingto comparison between the sound parameter and each threshold, accordingto the captured image of the imaging area; and superimposing the soundsource visual image corresponding to the sound parameter onto thecaptured image of the imaging area, on a per-predetermined-unit basis ofpixels which form the captured image of the imaging area, based on thederived sound parameter and the changed correspondence relationship anddisplaying the result on the monitor.

According to the disclosure, it is possible to present in detail, instages, the volume of the detected sound at the sound source position inthe imaging area of the camera device, regardless of the magnitude ofthe volume of the sound at the sound source position, and to assist theuser in accurately ascertaining the volume of the sound at the soundsource position.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of the schematicconfiguration of a pilotless flying object detection system of eachexemplary embodiment;

FIG. 2 is a diagram illustrating an example of the external appearanceof a sound source detection unit;

FIG. 3 is a block diagram illustrating an example of the internalconfiguration of a microphone array, in detail;

FIG. 4 is a block diagram illustrating an example of the internalconfiguration of an omnidirectional camera, in detail;

FIG. 5 is a block diagram illustrating an example of the internalconfiguration of a PTZ camera, in detail;

FIG. 6 is a block diagram illustrating an example of the internalconfiguration of a monitoring apparatus, in detail;

FIG. 7 is a timing chart illustrating an example of a detected soundsignal pattern of a pilotless flying object which is registered in amemory;

FIG. 8 is a timing chart illustrating an example of frequency variationin the detected sound signals which is obtained as a result of frequencyanalysis processing;

FIG. 9 is a sequence diagram illustrating an example of an operationprocedure of detecting a pilotless flying object and displaying adetection result thereof in a first exemplary embodiment;

FIG. 10 is a flowchart illustrating a detailed example of an operationprocedure of a pilotless flying object detection determination of stepS8 of FIG. 9;

FIG. 11 is a diagram illustrating an example of a situation in whichdirectivity setting directions in a monitoring area are sequentiallyscanned, and a pilotless flying object is detected;

FIG. 12 is a diagram illustrating an example of a display screen of afirst monitor in a case where a masking area is not set;

FIG. 13 is an explanatory diagram illustrating an example of a displayof the masking area at the time of performing an automatic learningprocess, in time series;

FIG. 14 is a sequence diagram illustrating an example of an operationprocedure of setting the masking area in the first exemplary embodiment;

FIG. 15 is a diagram illustrating an example of a display screen of thefirst monitor in a case where the masking area is set;

FIG. 16 is an explanatory diagram illustrating an outline of a dynamicchange of a display resolution of a sound pressure heat map in a secondexemplary embodiment;

FIG. 17 is a flowchart illustrating an example of an operation procedureof the dynamic change of the display resolution of the sound pressureheat map in the second exemplary embodiment;

FIG. 18 is an explanatory diagram illustrating an outline of widthadjustment of thresholds according to a frequency distribution of soundpressure values, and a display result of a captured image accompanied bythe width adjustment in the second exemplary embodiment;

FIG. 19A is an explanatory diagram illustrating an outline of a settingchange of inter-threshold widths defining the use of a sound sourcevisual image in the second exemplary embodiment;

FIG. 19B is an explanatory diagram illustrating an outline of thesetting change of the inter-threshold widths defining the use of thesound source visual image in the second exemplary embodiment;

FIG. 20 is an explanatory diagram illustrating an outline of a displayof the captured image accompanied by a setting change of inter-thresholdwidths defining the use of a crimson image and an ultramarine image inthe second exemplary embodiment;

FIG. 21 is a flowchart illustrating an example of an operation procedureof the setting change of the inter-threshold widths in the secondexemplary embodiment;

FIG. 22 is an explanatory diagram illustrating an outline of an overlaydisplay of an omnidirectional image and a translucent sound pressureheat map in a third exemplary embodiment;

FIG. 23 is a diagram illustrating an example of a display screen of afirst monitor on which the omnidirectional image and the translucentsound pressure heat map are overlay-displayed; and

FIG. 24 is a sequence diagram illustrating an example of an operationprocedure of the overlay display of the omnidirectional image and thetranslucent sound pressure heat map in the third exemplary embodiment.

DETAILED DESCRIPTION

Hereinafter, detailed description will be given of an embodiment(hereinafter referred to as the “exemplary embodiment”) whichspecifically discloses a pilotless flying object detection system and apilotless flying object detection method for detecting a pilotlessflying object (for example, drone or radio controlled helicopter) as amonitoring target, as an example of a monitoring system or a monitoringmethod executed in the monitoring system according to the disclosure,with reference to the diagrams, as appropriate. Description in greaterdetail than is necessary may be omitted. For example, detaileddescription of matters which are already well known, and duplicatedescription of configurations which are effectively the same may beomitted. This is in order to avoid rendering the following descriptionunnecessarily verbose, and to facilitate understanding of a personskilled in the art. The attached diagrams and the following descriptionare provided in order for a person skilled in the art to sufficientlyunderstand the disclosure, and are not intended to limit the scope ofthe claims.

Hereinafter, a user of a pilotless flying object detection system (forexample, a surveillance worker who patrols and guards the monitoringarea) is simply referred to as a “user”.

FIG. 1 is a diagram illustrating an example of the schematicconfiguration of pilotless flying object detection system 5 of eachexemplary embodiment. Pilotless flying object detection system 5 detectspilotless flying object do (for example, refer to FIG. 14) which is atarget of the user as a detection target. Pilotless flying object do isa drone which flies autonomously using a global positioning system (GPS)function for example, a radio controlled helicopter which is wirelesslycontrolled by a third party, or the like. Pilotless flying object dn isused in aerial photography of a target, delivery of goods, or the like,for example.

In each exemplary embodiment, a multi-copter drone on which a pluralityof rotors (in other words, rotary blades) are installed is exemplifiedas pilotless flying object dn. In a multi-copter drone, generally, in acase in which there are two rotor blades, a high frequency wave of twicethe frequency of a specific frequency, and further, a high frequencywave of a multiple frequency thereof are generated. Similarly, in a casein which there are three rotor blades, a high frequency wave of threetimes the frequency of a specific frequency, and further, a highfrequency wave of a multiple frequency thereof are generated. The sameapplies to a case in which the number of rotor blades is greater than orequal to four.

Pilotless flying object detection system 5 is configured to include aplurality of sound source detection units UD1, . . . , UDk, . . . , andUDn, monitoring apparatus 10, first monitor MN1, second monitor MN2, andrecorder RC. The plurality of sound source detection units UD aremutually connected to monitoring apparatus 10 via network NW. k is anatural number of 1 to n. Each sound source detection unit, for example,sound source detection unit UD1 is configured to include microphonearray MA1, omnidirectional camera CA1, and PTZ camera CZ1, and othersound source detection units UDk have the same configuration. Except forcases in which it is necessary to particularly distinguish theindividual sound source detection units, these will be referred to assound source detection unit UDk or, simply sound source detection unitUD. Similarly, except for cases in which it is necessary to particularlydistinguish the individual microphone arrays, omnidirectional cameras,and PTZ cameras, these will be referred to as microphone array MAk orMA, omnidirectional camera CAk or CA, and PTZ camera CZk or CZ.

In sound source detection unit UDk, microphone array MAk collects soundof all directions in a sound collection area in which the device isinstalled (for example, the monitoring area as the monitoring target) ina non-directional state.

Microphone array MAk includes body 15 (refer to FIG. 2) in the center ofwhich a cylindrical opening of a predetermined width is formed. Examplesof sounds which are used as sound collection targets of microphone arrayMAk include mechanical operating sound of a drone or the like,vocalizations uttered by a human or the like, and a wide variety ofother sounds, including not only sounds of an audible frequency (thatis, 20 Hz to 23 kHz) domain, but also low frequency sounds which arelower than audible frequencies and ultrasonic sounds which exceedaudible frequencies.

Microphone array MAk includes a plurality of non-directional microphonesM1 to Mq (refer to FIG. 3). q is a natural number greater than or equalto 2. Microphones M1 to Mq are disposed at a predetermined interval (forexample, a uniform interval) in a coaxial circular shape along acircumferential direction around the opening which is provided in body15. Electret Condenser Microphones (ECM) are used for the microphones,for example. Microphone array MAk transmits the sound data of the soundobtained through collection by microphones M1 to Mq (refer to laterdescription) to monitoring apparatus 10 via network NW. The arrangementof microphones M1 to Mq described above is an example, and otherarrangements (for example, arrangements in a square shape, arrangementin a rectangular shape) may be adopted, but it is preferable thatmicrophones M1 to Mq are arranged side by side at equal intervals.

Microphone array MAk includes a plurality of microphones M1 to Mq (forexample, q=32), and a plurality of amplifiers PA1 to PAq (refer to FIG.3) which amplify the output signals of the plurality of microphones M1to Mq, respectively. The analog signals which are output from eachamplifier are converted to corresponding digital signals by A/Dconverters A1 to Aq which are described later (refer to FIG. 3). Thenumber of microphones in the microphone array MAk is not limited to 32,and may be another number (for example, 16, 64, or 128).

Omnidirectional camera CAk which has approximately the same volume asthe opening is housed inside the opening which is formed in the middleof body 15 (refer to FIG. 2) of microphone array MAk. In other words,microphone array MAk and omnidirectional camera CAk are disposedintegrally such that body centers thereof are in the same axis direction(refer to FIG. 2). Omnidirectional camera CAk is a camera on whichfish-eye lens 45 a (refer to FIG. 4), which is capable of capturing anomnidirectional image of the monitoring area as the imaging area ofomnidirectional camera CAk which is the sound collection area, ismounted. In each exemplary embodiment, description is given assumingthat the sound collection area of microphone array MAk and the imagingarea of omnidirectional camera CAk are a shared monitoring area;however, the spatial sizes (for example, volume) of the sound collectionarea and the imaging area may not be the same. For example, the volumeof the sound collection area may be larger or smaller than the volume ofthe imaging area. In other words, it is sufficient for the soundcollection area and the imaging area to have a shared volume portion.Omnidirectional camera CAk functions as a monitoring camera capable ofimaging the imaging area in which sound source detection unit UDk isinstalled, for example. In other words, omnidirectional camera CA has anangle of view of 180° in the vertical direction and 360° in thehorizontal direction, and images monitoring area 8 (refer to FIG. 11)which is a hemisphere, for example, as the imaging area.

In each sound source detection unit UDk, omnidirectional camera CAk andmicrophone array MAk are disposed coaxially due to omnidirectionalcamera CAk being fitted inside the opening of body 15. In this manner,due to the optical axis of omnidirectional camera CAk and the centeraxis of the body of microphone array MAk matching, the imaging area andthe sound collection area match substantially in the axial circumferencedirection (that is, the horizontal direction), and it becomes possibleto express the position of an object in the image and the position of asound source of a sound collection target in the same coordinate system(for example, coordinates indicated by (horizontal angle, verticalangle)). Each sound source detection unit UDk is attached such thatupward in the vertical direction becomes a sound collection surface andan imaging surface, for example, in order to detect pilotless flyingobject dn which flies from the sky (refer to FIG. 2).

Monitoring apparatus 10 is configured using a personal computer (PC) ora server, for example. Monitoring apparatus 10 is capable of formingdirectionality (that is, beam forming) in relation to the sound of allDirections which is collected by microphone array MAk using an arbitrarydirection as a main beam direction based on a user operation, andemphasizing the sound of the directivity setting direction.

Monitoring apparatus 10 uses the image (hereinafter, this may beshortened to “captured image”) which is captured by omnidirectionalcamera CAk and processes the captured image to generate anomnidirectional image. The omnidirectional image may be generated byomnidirectional camera CAk instead of monitoring apparatus 10.

Monitoring apparatus 10 superimposes an image (refer to FIG. 15) of thesound pressure heat map based on the calculated value of the soundparameter (for example, sound pressure described below) specifying thevolume of the sound which is collected by microphone array MAk, onto thecaptured image which is captured by omnidirectional camera CAk, andoutputs the result to first monitor MN1 to be displayed.

Monitoring apparatus 10 may display a visual image (for example,identification mark) by which it is easy for a user to visuallydetermine detected pilotless flying object dn, on omnidirectional imageIMG1, at a position of pilotless flying object dn of first monitor MN1.For example, the visual information means information which is displayedon omnidirectional image IMG1 so as to be clearly distinguished fromother objects when the user views omnidirectional image IMG1, and thesame is applied to the description below.

First monitor MN1 displays omnidirectional image IMG1 which is capturedby omnidirectional camera CAk. Second monitor MN2 displaysomnidirectional image IMG2 which is captured by omnidirectional cameraCAk. First monitor MN1 generates a composite image obtained bysuperimposing the identification mark onto omnidirectional image IMG1and displays the composite image. In FIG. 1, two monitors of firstmonitor MN1 and second monitor MN2 are connected to monitoring apparatus10, but only first monitor MN1 may be connected to monitoring apparatus10. Either first monitor MN1 or second monitor MN2, or both firstmonitor MN1 and second monitor MN2 may be configured as an integralapparatus with monitoring apparatus 10.

Recorder RC is configured, for example, using a hard disk drive or asemiconductor memory such as a flash memory, and stores data (refer tolater description) of various images generated by monitoring apparatus10, or various data of the omnidirectional image or audio transmittedfrom each sound source detection unit UDk. Recorder RC may be configuredas an integral apparatus with monitoring apparatus 10 or may be omittedfrom the configuration of pilotless flying object detection system 5.

In FIG. 1, the plurality of sound source detection units UDk andmonitoring apparatus 10 have a communication interface, and areinterconnected via network NW to be capable of data communication.Network NW may be a wired network (for example, an intranet, theInternet, or a wired local area network (LAN)), and may be a wirelessnetwork (for example, a wireless LAN). Sound source detection units UDkand monitoring apparatus 10 may be connected directly without connectingvia network NW. All of monitoring apparatus 10, first monitor MN1,second monitor MN2, and recorder RC are installed in monitoring room RMin which a user such as a surveillance worker resides at the time ofmonitoring.

FIG. 2 is a diagram illustrating the external appearance of sound sourcedetection unit UD. In addition to microphone array MA, omnidirectionalcamera CA, and PTZ camera CZ described earlier, sound source detectionunit UD includes supporting stand 70 which mechanically supports theearlier-described elements. Supporting stand 70 has a structurecombining tripod 71, two rails 72 which are fixed to top board 71 a oftripod 71, and first adapter plate 73 and second adapter plate 74 whichare attached to the end of each of two rails 72.

First adapter plate 73 and second adapter plate 74 are attached tostraddle two rails 72, and have substantially the same planar surfaces.First adapter plate 73 and second adapter plate 74 slide freely on tworails 72, and are fixed adjusted to positions separated from or proximalto each other.

First adapter plate 73 is a disc-shaped plate member. Opening 73 a isformed in the center of first adapter plate 73. Body 15 of microphonearray MA is housed and fixed in opening 73 a. Meanwhile, second adapterplate 74 is a substantially rectangular plate member. Opening 74 a isformed in a portion close to the outside of second adapter plate 74. PTZcamera CZ is housed and fixed in opening 74 a.

As illustrated in FIG. 2, in the initial installation state, opticalaxis L1 of omnidirectional camera CA which is housed in body 15 ofmicrophone array MA and optical axis L2 of PTZ camera CZ which isattached to second adapter plate 74 are set to be parallel to eachother.

Tripod 71 is supported on a ground surface by three legs 71 b, freelymoves the position of top board 71 a in the vertical direction inrelation to the ground surface through manual operation, and is capableof adjusting the orientation of top board 71 a in the pan direction andthe tilt direction. Accordingly, it is possible to set the soundcollection area of microphone array MA (in other words, the imaging areaof omnidirectional camera CA or the monitoring area of pilotless flyingobject detection system 5) to an arbitrary orientation.

FIG. 3 is a block diagram illustrating an example of the internalconfiguration of microphone array MAk, in detail. Microphone array MAkillustrated in FIG. 3 is configured to include a plurality ofmicrophones M1 to Mq (for example, q=32), a plurality of amplifiers PA1to PAq, a plurality of A/D converters A1 to Aq, audio data processor 25,and transmitter 26. The plurality of amplifiers PA1 to PAq amplify thecorresponding output signals of the plurality of microphones M1 to Mq,and the plurality of A/D converters A1 to Aq convert the analog signalswhich are output from amplifiers PA1 to PAq into corresponding digitalsignals.

Audio data processor 25 generates sound data packets based on thedigital audio signals which are output from A/D converters A1 to Aq.Transmitter 26 transmits the audio data packets which are generated byaudio data processor 25 to monitoring apparatus 10 via network NW.

In this manner, microphone array MAk amplifies the output signals ofmicrophones M1 to Mq using amplifiers PA1 to PAq, and converts theamplified signals into digital audio signals using A/D converters A1 toAq. Subsequently, microphone array MA generates audio data packets usingaudio data processor 25, and transmits the audio data packets tomonitoring apparatus 10 via network NW.

FIG. 4 is a block diagram illustrating an example of the internalconfiguration of omnidirectional camera CAk, in detail. Omnidirectionalcamera CAk illustrated in FIG. 4 is configured to include CPU 41,transceiver 42, power supply manager 44, image sensor 45, memory 46, andnetwork connector 47. Fish-eye lens 45 a is provided on the front stage(that is, the right side in FIG. 4) of image sensor 45.

CPU 41 performs signal processing for performing overall control of theoperations of the elements of omnidirectional camera CAk, input-outputprocessing of data with other elements, computational processing ofdata, and storage processing of data. Instead of CPU 41, a processorsuch as a micro processing unit (MPU) or a digital signal processor(DSP) may be provided.

For example, CPU 41 generates cut-out image data which is obtained bycutting out an image of a specific range (direction) within theomnidirectional image data by the designation of a user operatingmonitoring apparatus 10, and saves the generated image data in memory46.

Image sensor 45 is configured using a complementary metal-oxidesemiconductor (CMOS) sensor or a charge coupled device (CCD) sensor, andacquires omnidirectional image data by subjecting an optical image of anobject in an imaging area, which is formed by fish-eye lens 45 a toimage processing on a light receiving surface.

Memory 46 includes ROM 46 z, RAM 46 y, and memory card 46 x. Programsand setting value data for defining the operations of omnidirectionalcamera CAk are stored in ROM 46 z, RAM 46 y stores omnidirectional imagedata or cut-out image data which is obtained by cutting out a portionrange of the omnidirectional image data, and work data, and memory card46 x is connected to omnidirectional camera CAk to be freely insertedand removed, and stores various data.

Transceiver 42 is a network interface (I/F) which controls datacommunication with network NW to which transceiver 42 is connected vianetwork connector 47.

Power supply manager 44 supplies direct current power to the elements ofomnidirectional camera CA. Power supply manager 44 may supply directcurrent power to devices which are connected to network NW via networkconnector 47.

Network connector 47 is a connector which transmits omnidirectionalimage data or two-dimensional panorama image data to monitoringapparatus 10 via network NW, and is capable of supplying power via anetwork cable.

FIG. 5 is a block diagram illustrating an example of the internalconfiguration of PTZ camera CZk, in detail. Description of the sameelements as in omnidirectional camera CAk will be omitted by assigningreference signs corresponding to the elements in FIG. 4. PTZ camera CZkis a camera capable of adjusting the optical axis direction (alsoreferred to as the imaging direction) through angle of view changeinstructions from monitoring apparatus 10.

In the same manner as omnidirectional camera CAk, PTZ camera CZkincludes CPU 51, transceiver 52, power supply manager 54, image sensor55, imaging lens 55 a, memory 56, and network connector 57, andadditionally includes imaging direction controller 58 and lens drivingmotor 59. If an angle of view change instruction of monitoring apparatus10 is present, CPU 51 notifies imaging direction controller 58 of theangle of view change instruction.

In accordance with the angle of view change instruction of which imagingdirection controller 58 is notified by CPU 51, imaging directioncontroller 58 controls the imaging direction of PTZ camera CZk in atleast one of the pan direction and the tilt direction, and further, asnecessary, outputs a control signal for changing the zoom ratio to lensdriving motor 59. In accordance with the control signal, lens drivingmotor 59 drives imaging lens 55 a, changes the imaging direction of theimaging lens (the direction of optical axis L2 illustrated in FIG. 2),and adjusts the focal length of imaging lens 55 a to change the zoomratio.

Imaging lens 55 a is configured using one lens, or two or more lenses.In imaging lens 55 a, the optical axis direction of the pan rotation andthe tilt rotation is changed by the driving of lens driving motor 59according to the control signal from imaging direction controller 58.

FIG. 6 is a block diagram illustrating an example of the internalconfiguration of monitoring apparatus 10, in detail. Monitoringapparatus 10 illustrated in FIG. 6 includes at least transceiver 31,console 32, signal processor 33, speaker (SPK) 37, memory 38, andsetting manager 39.

Transceiver 31 receives the omnidirectional image data or the cut-outvideo data which is transmitted by omnidirectional camera CAk, and theaudio data which is transmitted by microphone array MAk, and outputs thereceived data to signal processor 33.

Console 32 is a user interface (UI) for notifying signal processor 33 ofthe content of an input operation of the user, and is configured by apointing device such as a mouse and a keyboard. Console 32 may beconfigured using a touch panel or a touch pad which is disposedcorresponding to each screen of first monitor MN1 and second monitorMN2, for example, and with which direct input operation is possiblethrough a finger or a stylus pen of the user.

In a case where in first monitor MN1 and second monitor MN2, red areaRD1 of the sound pressure heat map (refer to FIG. 15) which issuperimposed to be displayed on the captured image (omnidirectionalimage IMG1) of any omnidirectional camera CAk is designated by a user,console 32 acquires coordinate data which indicates the designatedposition, to output the coordinate data to signal processor 33. Signalprocessor 33 reads the sound data collected by microphone array MAkcorresponding to omnidirectional camera CAk from memory 38, formsdirectionality in the direction toward the sound source positioncorresponding to the designated position from microphone array MAk, andsubsequently outputs the directionality to speaker 37. Accordingly, theuser is capable of clearly ascertaining in a state in which the sound atnot only pilotless flying object do but also other positions designatedon the captured image (omnidirectional image IMG1) by the user isemphasized.

Signal processor 33 is configured using a central processing unit (CPU),a micro processing unit (MPU), or a digital signal processor (DSP), forexample, and performs control processing for performing overall controlof the operation of the elements of monitoring apparatus 10,input-output processing of data with other elements, computational(calculation) processing of data, and storage processing of data. Signalprocessor 33 includes directivity processor 63, frequency analyzer 64,object detector 65, detection result determiner 66, scanning controller67, detecting direction controller 68, masking area setter 69 a,threshold adjuster 69 b, sound source direction detector 34, and outputcontroller 35. Monitoring apparatus 10 is connected to first monitor MN1and second monitor MN2.

Sound source direction detector 34 estimates the sound source positionusing the audio data of the audio of monitoring area 8 which iscollected by microphone array MAk according to a well-known cross-powerspectrum phase analysis (CSP) method. In the CSP method, when soundsource direction detector 34 divides monitoring area 8 illustrated inFIG. 11 into a plurality of blocks and sound is collected by microphonearray MA, sound source direction detector 34 is capable of approximatelyestimating the sound source position in monitoring area 8 by determiningwhether or not a sound exceeding a threshold sound pressure, soundvolume or the like is present on a per-block basis.

In addition, sound source direction detector 34 as the sound parameterderiving unit calculates the sound pressure as the sound parameter, on aper-pixel basis using the individual pixels which form theomnidirectional image data of monitoring area 8 based on theomnidirectional image data which is captured by omnidirectional cameraCAk and the audio data which is collected by microphone array MAk. Soundsource direction detector 34 outputs a calculated value as thecalculation result of the sound pressure, to output controller 35.

Setting manager 39 includes, in advance, a coordinate transformationequation relating to the coordinates of a position designated by theuser in relation to the screen of first monitor MN1 on which theomnidirectional image data which is captured by omnidirectional cameraCAk is displayed. The coordinate transformation equation is an equationfor transforming the coordinates (that is, (horizontal angle, verticalangle)) of a user-designated position in the omnidirectional image datainto coordinates of a direction viewed from PTZ camera CZ based on adifference in the physical distance between the installation position ofomnidirectional camera CAk (refer to FIG. 2) and the installationposition of PTZ camera CZk (refer to FIG. 2).

Signal processor 33 uses the coordinate transformation equation held bysetting manager 39 to calculate the coordinates (θMAh, θMAv) indicatingthe directivity setting direction facing the actual sound sourceposition corresponding to the position designated by the user from theinstallation position of PTZ camera CZk, using the installation positionof PTZ camera CZk (refer to FIG. 2) as a reference. θMAh is thehorizontal angle of a direction facing the actual sound source positioncorresponding to the position designated by the user, from theperspective of the installation position of PTZ camera CZk. θMAv is thevertical angle of a direction facing the actual sound source positioncorresponding to the position designated by the user, from theperspective of the installation position of PTZ camera CZk. Asillustrated in FIG. 2, the distance between omnidirectional camera CAkand PTZ camera CZk is known, and since optical axes L1 and L2 areparallel to each other, it is possible to realize the calculationprocess of the coordinate transformation equation using a well-knowngeometric computation, for example. The sound source position is theactual sound source position corresponding to the position designatedfrom console 32 by an operation of a finger or a stylus pen of the userin relation to the video data which is displayed on first monitor MN1and second monitor MN2.

As illustrated in FIG. 2, omnidirectional camera CAk and microphonearray MAk are both disposed coaxially with the optical axis direction ofomnidirectional camera CAk and the center axis of the body of microphonearray MAk in the exemplary embodiment. Therefore, the coordinates of thedesignated position derived by omnidirectional camera CAk according tothe designation of the user in relation to first monitor MN1 on whichthe omnidirectional image data is displayed may be treated as the sameas the emphasized direction (also referred to as the directivity settingdirection) of the sound from the perspective of microphone array MAk. Inother words, when user designation in relation to first monitor MN1 (orsecond monitor MN2 is possible) on which the omnidirectional image datais displayed is present, monitoring apparatus 10 transmits thecoordinates of the designated position in the omnidirectional image datato omnidirectional camera CAk. Accordingly, omnidirectional camera CAkcalculates the coordinates (horizontal angle, vertical angle) indicatingthe direction of the sound source position corresponding to thedesignated position from the perspective of omnidirectional camera CAkusing the coordinates of the designated position which are transmittedfrom monitoring apparatus 10. Omnidirectional camera CAk transmits thecalculation results of the coordinates indicating the direction of thesound source position to monitoring apparatus 10. Monitoring apparatus10 is capable of using the coordinates (horizontal angle, verticalangle) which are calculated by omnidirectional camera CAk as thecoordinates (horizontal angle, vertical angle) indicating the directionof the sound source position from the perspective of microphone arrayMAk.

However, in a case in which omnidirectional camera CAk and microphonearray MAk are not disposed coaxially, it is necessary for settingmanager 39 to follow the method described in Japanese Patent UnexaminedPublication No. 2015-029241 to transform the coordinates derived byomnidirectional camera CAk into the coordinates of the direction fromthe perspective of microphone array MAk.

Setting manager 39 holds first threshold th1, second threshold th2, andthird threshold th3 (for example, refer FIG. 9) which are compared tosound pressure p on a per-pixel basis using the pixels which form theomnidirectional image data or two-dimensional panorama image datacalculated by sound source direction detector 34. Here, sound pressure pis used as an example of a sound parameter relating to the sound source,represents the magnitude of the sound which is collected by microphonearray MA, and is differentiated from the sound volume which representsthe magnitude of the sound which is output from speaker 37. Firstthreshold th1, second threshold th2, and third threshold th3 are valueswhich are compared to the sound pressure of the sound which is generatedin monitoring area 8, and are set to predetermined values fordetermining the sound emitted by pilotless flying object dn, forexample. It is possible to set a plurality of thresholds other thanfirst threshold th1, second threshold th2, and third threshold th3, andhere, in order for simple description, first threshold th1, secondthreshold th2 which is a larger value than first threshold th1, thirdthreshold th3 which is a larger value than those two thresholds are set,totaling three thresholds (first threshold th1<second thresholdth2<third threshold th3).

As described later, in the sound pressure heat map generated by outputcontroller 35, red area RD1 (refer to FIG. 15) of the pixels at which agreater sound pressure than third threshold th3 is obtained is renderedin red, for example, on first monitor MN1 on which the omnidirectionalimage data is displayed. Pink area PD1 of the pixels at which a soundpressure which is greater than second threshold th2 and less than orequal to third threshold th3 is obtained is rendered in pink, forexample, on first monitor MN1 on which the omnidirectional image data isdisplayed. Blue area BD1 of the pixels at which a sound pressure whichis greater than first threshold th1 and less than or equal to secondthreshold th2 is obtained is rendered in blue, for example, on firstmonitor MN1 on which the omnidirectional image data is displayed. AreaN1 of the pixels having a sound pressure less than or equal to firstthreshold th1 is rendered colorless, for example, on first monitor MN1on which the omnidirectional image data is displayed, that is, is nodifferent from the display color of the omnidirectional image data.

Speaker 37 outputs the audio data collected by microphone array MAk, orthe audio data which is collected by microphone array MAk and for whichdirectionality is formed by signal processor 33. Speaker 37 may beconfigured as a separate device from monitoring apparatus 10.

Memory 38 is configured using a ROM or a RAM. Memory 38 holds variousdata including sound data of a fixed zone, setting information,programs, and the like, for example. Memory 38 includes pattern memoryin which sound patterns which are characteristic to the individualpilotless flying objects do are registered. Furthermore, memory 38stores data of the sound pressure heat map generated by outputcontroller 35. An identification mark which schematically represents theposition of pilotless flying object dn is registered in memory 38. Theidentification mark which is used here is a star-shaped symbol as anexample. The identification mark is not limited to a star shape, and inaddition to a circle shape or a rectangle shape, may further be a symbolor character such as a fylfot which is reminiscent of a pilotless flyingobject. The display form of the identification mark may be changedbetween day and night, for example, a star shape during the day, and arectangular shape during the night so as not to be confused for a star.The identification mark may be dynamically changed. For example, astar-shaped symbol may be displayed in a blinking manner, or may berotated, further engaging the attention of the user.

FIG. 7 is a timing chart illustrating an example of a detected soundpattern of pilotless flying object dn which is registered in memory 38.The detected sound pattern illustrated in FIG. 7 is a combination offrequency patterns, and includes sounds of four frequencies f1, f2, f3,and f4 which are generated by the rotation of four rotors which areinstalled on the multi-copter pilotless flying object dn, or the like.The signals of the frequencies are signals of frequencies of differentsounds which are generated in accordance with the rotation of aplurality of blades which are axially supported on each rotor, forexample.

In FIG. 7, the frequency areas shaded with diagonal lines are areas withhigh sound pressure. The detected sound pattern may include not only thenumber of sounds and the sound pressure of the plurality of frequencies,but also other sound information. For example, a sound pressure raterepresenting the sound pressure ratio of the frequencies or the like isexemplified. Here, for example, the detection of pilotless flying objectdn is determined according to whether or not the sound pressure of eachfrequency contained in the detected sound pattern exceeds a threshold.

Directivity processor 63 uses the sound signals (also referred to assound data) which are collected by the non-directional microphones M1 toMq, performs a directionality forming process described earlier (beamforming), and performs an extraction process of the sound data in whichdirections of other areas except for the masking area set by maskingarea setter 69 a are used as the directivity setting direction.Directivity processor 63 is also capable of performing an extractionprocess of the sound data in which a direction range of other areasexcept for the masking area set by masking area setter 69 a is used as adirectivity setting area. Here, the directivity setting area is a rangeincluding a plurality of adjacent directivity setting directions, and incomparison to the directivity setting direction, is intended to includea degree of spreading in the directivity setting direction.

Frequency analyzer 64 performs frequency analysis processing on thesound data which is subjected to the extraction process in thedirectivity setting direction by directivity processor 63. In thefrequency analysis processing, the frequency and the sound pressurethereof included in the sound data of the directivity setting directionare detected.

FIG. 8 is a timing chart illustrating an example of frequency variationin the detected sound signals which is obtained as a result of thefrequency analysis processing. In FIG. 8, four frequencies f11, f12,f13, and f14, and the sound pressure of each frequency are obtained asthe detected sound signals (that is, the detected sound data). In FIG.8, the fluctuation in each frequency which changes irregularly occursdue to fluctuations in the rotation of the rotors (the rotary blades)which change slightly when pilotless flying object dn controls theposture of the body of pilotless flying object dn.

Object detector 65 as a detector performs a detection process ofpilotless flying object dn by using the result of the frequency analysisprocessing of frequency analyzer 64. Specifically, in the detectionprocess of pilotless flying object dn, object detector 65 compares thedetected sound pattern which is obtained as a result of the frequencyanalysis processing (refer to FIG. 8) (frequencies f11 to f14), to thedetected sound pattern which is registered in advance in the patternmemory of memory 38 (refer to FIG. 7) (frequencies f1 to f4) in otherareas except for the masking area set by masking area setter 69 a.Object detector 65 determines whether or not both of the patterns ofdetected sounds are similar.

Whether or not both of the patterns of detected sounds are similar isdetermined as follows, for example. In a case in which the soundpressures of at least two frequencies contained in the detected sounddata of four frequencies f1, f2, f3, and f4 exceed a threshold, objectdetector 65 determines the sound patterns to be similar and detectspilotless flying object dn. Pilotless flying object dn may be detectedin a case in which other conditions are satisfied.

In a case in which detection result determiner 66 determines thatpilotless flying object dn is not present, detection result determiner66 instructs detecting direction controller 68 to transition todetecting pilotless flying object dn in the next directivity settingdirection. In a case in which detection result determiner 66 determinesthat pilotless flying object dn is present as a result of the scanningof the directivity setting direction, detection result determiner 66notifies output controller 35 of the detection results of pilotlessflying object dn. Information of the detected pilotless flying object dnis included in the detection results. The information of pilotlessflying object dn includes identification information of pilotless flyingobject dn, and positional information (for example, directioninformation) of pilotless flying object dn in the sound collection area.

Detecting direction controller 68 controls the direction for detectingpilotless flying object dn in the sound collection area based on theinstructions from detection result determiner 66. For example, detectingdirection controller 68 sets an arbitrary direction of directivitysetting area BF1 which contains the sound source position which isestimated by sound source direction detector 34 in the entirety of thesound collection area as the detection direction.

Scanning controller 67 instructs directivity processor 63 to performbeam forming using the detection direction which is set by detectingdirection controller 68 as the directivity setting direction.

Directivity processor 63 performs beam forming on the directivitysetting direction which is instructed from scanning controller 67. Inthe initial settings, directivity processor 63 uses the initial positionin directivity setting area BF1 (refer to FIG. 11) which includes thesound source position which is estimated by sound source directiondetector 34 as directivity setting direction BF2. Directivity settingdirection BF2 is set successively from within directivity setting areaBF1 by detecting direction controller 68.

Masking area setter 69 a sets the masking area to be excluded from thedetection of pilotless flying object dn, which appears in theomnidirectional image or the two-dimensional panorama image (that is,the captured image) based on the omnidirectional image data or thetwo-dimensional panorama image data of monitoring area 8 captured byomnidirectional camera CAk, and the audio data of monitoring area 8collected by microphone array MAk. The setting of the masking area willbe described later in detail with reference to FIGS. 13 and 14.

Output controller 35 controls the operations of first monitor MN1,second monitor MN2, and speaker 37, outputs the omnidirectional imagedata or the two-dimensional panorama image data which is transmittedfrom omnidirectional camera CAk to first monitor MN1 and second monitorMN2 to be displayed, and further outputs the audio data which istransmitted from microphone array MAk to speaker 37. In a case in whichpilotless flying object do is detected, output controller 35 outputs theidentification mark which represents pilotless flying object do to firstmonitor MN1 (or second monitor MN2 is possible) in order to superimposethe identification mark onto omnidirectional image and display theresult.

Output controller 35 subjects the sound data of the directivity settingdirection to emphasis processing by using the audio data which iscollected by microphone array MAk and the coordinates which indicate thedirection of the sound source position which is derived byomnidirectional camera CAk to perform a directionality forming processon the sound data which is collected by microphone array MAk.

Output controller 35 generates a sound pressure heat map in which acalculated value of the sound pressure is allocated to the position of apixel on a per-pixel basis using the individual pixels which form theomnidirectional image data or two-dimensional panorama image data, byusing the sound pressure values on a per-pixel basis using the pixelswhich form the omnidirectional image data or two-dimensional panoramaimage data which are calculated by sound source direction detector 34.Furthermore, output controller 35 generates the sound pressure heat mapsuch as that illustrated in FIG. 15 by performing a color transformationprocess on the sound pressure values on a per-pixel basis using thepixels of the generated sound pressure heat map such that it is easy fora user to perform visual determination.

Output controller 35 is described as generating a sound pressure heatmap in which sound pressure values which are calculated in pixel unitsare allocated to corresponding pixel positions; however, the soundpressure heat map may be generated by calculating the average value ofthe sound pressure values in pixel block units formed of a predeterminednumber of (for example, 2×2, 4×4) pixels without calculating the soundpressure on a per-pixel basis, and allocating the average value of thesound pressure values corresponding to the corresponding predeterminednumber of pixels.

The details of threshold adjuster 69 b will be described in a secondexemplary embodiment described below, and thus the detailed descriptionthereof is omitted here.

Next, the operation of pilotless flying object detection system 5 in theexemplary embodiment will be described in detail.

FIG. 9 is a sequence diagram illustrating an example of an operation ofdisplaying a detection result of pilotless flying object dn in pilotlessflying object detection system 5 of the first exemplary embodiment. Whenpower is input to the devices (for example, first monitor MN1,monitoring apparatus 10, omnidirectional camera CAk, and microphonearray MAk) of pilotless flying object detection system 5, pilotlessflying object detection system 5 starts operating. In the description ofFIG. 9, it is assumed that the masking area to be excluded from thedetection of pilotless flying object dn is already set, and informationindicating the masking area is registered in memory 38.

In the initialization operations, monitoring apparatus 10 performs animage transmission request in relation to omnidirectional camera CAk(S1). Omnidirectional camera CAk starts the imaging processcorresponding to the input of power in accordance with the request.Furthermore, monitoring apparatus 10 performs a sound transmissionrequest in relation to microphone array MAk (S2). Microphone array MAkstarts the sound collection process corresponding to the input of powerin accordance with the request.

Once the initialization operations are completed, omnidirectional cameraCAk transmits the data of the captured image (for example, a still imageor a video) which is obtained through imaging to monitoring apparatus 10via network NW (S3). In order for the brief description, in FIG. 9, itis described that the omnidirectional image data is transmitted fromomnidirectional camera CAk; however, two-dimensional panorama image datamay be transmitted, and the same is applied to FIG. 14. Monitoringapparatus 10 converts the captured image data which is transmitted fromomnidirectional camera CAk into display data such as NTSC, outputs thedisplay data to first monitor MN1, and instructs first monitor MN1 todisplay the omnidirectional image data (S4). When the display datatransmitted from monitoring apparatus 10 is input, first monitor MN1displays on the screen, the data (refer to FIGS. 12 and 15) ofomnidirectional image IMG1 by omnidirectional camera CAk.

Microphone array MAk encodes the sound data of the sound obtainedthrough collection and transmits the encoded sound data to monitoringapparatus 10 via network NW (S5). In monitoring apparatus 10, soundsource direction detector 34 calculates the sound pressure as the soundparameter, on a per-pixel basis using the individual pixels which formthe omnidirectional image data of monitoring area 8 based on theomnidirectional image data which is captured by omnidirectional cameraCAk and the audio data which is collected by microphone array MAk, andfurther estimates the sound source position within monitoring area 8(S6). When monitoring apparatus 10 detects pilotless flying object dn,the estimated sound source position is used as the reference position ofdirectivity setting area BF1 which is necessary for the initial settingof the directivity setting direction.

In addition, in monitoring apparatus 10, output controller 35 generatesa sound pressure map in which a calculated value of the sound pressureis allocated to the position of a pixel on a per-pixel basis using thepixels which form the omnidirectional image data, by using the soundpressure values on a per-pixel basis using the pixels which form theomnidirectional image data which are calculated by sound sourcedirection detector 34. Furthermore, output controller 35 generates asound pressure heat map such as that illustrated in FIG. 15 byperforming a color transformation process on the sound pressure valueson a per-pixel basis using the pixels of the generated sound pressuremap so as to obtain the visual image (for example, colored image) suchthat it is easy for the user to perform visual determination (S7).

Further, when signal processor 33 forms sequential directionality forthe area other than the masking area set by masking area setter 69 a, byusing the audio data transmitted from microphone array MAk in step S5,monitoring apparatus 10 performs detection determination of pilotlessflying object dn for each directivity setting direction in which thedirectionality is formed (S8). The detection determination process ofpilotless flying object dn will be described later in detail withreference to FIGS. 10 and 11.

In a case in which pilotless flying object dn is detected as a result ofthe detection determination process, output controller 35 in monitoringapparatus 10 superimposes the sound pressure heat map generated in stepS7, and the identification mark, which represents pilotless flyingobject dn which is present in the directivity setting direction detectedin step S8, onto omnidirectional image IMG1 which is displayed on thescreen of first monitor MN1 and displays the result (S9).

First monitor MN1 combines (superimposes) the sound pressure heat map onomnidirectional image IMG1 according to the instruction from monitoringapparatus 10 and displays the result, and combines (superimposes) theidentification mark representing pilotless flying object dn onomnidirectional image IMG1 and displays the result (S10). Subsequently,the process of pilotless flying object detection system 5 returns tostep S3, and processes of steps S3 to S10 are repeated until apredetermined event such as the power being operated to turn off, forexample, is detected.

FIG. 10 is a flowchart illustrating a detailed example of a pilotlessflying object detection determination procedure of step S8 of FIG. 9. Insound source detection unit UDk, directivity processor 63 setsdirectivity setting area BF1 which is other than the masking area and isbased on the sound source position estimated by sound source directiondetector 34, as the initial position of directivity setting directionBF2 by using information of the masking area set by masking area setter69 a (S21). The information of the masking area is coordinates in adirection toward the masking area seen from microphone array MAk.

FIG. 11 is a diagram illustrating an example of a situation in whichdirectivity setting directions BF2 in monitoring area 8 are sequentiallyscanned, and pilotless flying object do is detected. The initialposition is preferably other than the masking area set by masking areasetter 69 a and may not be limited to directivity setting area BF1 basedon the sound source position of monitoring area 8 which is estimated bysound source direction detector 34. That is, an arbitrary positiondesignated by the user may be set as the initial position as long as theposition is other than the masking area set by masking area setter 69 a,and the inside of monitoring area 8 may be sequentially scanned. Due tothe initial position not being limited, even in a case in which thesound source included in directivity setting area BF1 based on theestimated sound source position is not a pilotless flying object, itbecomes possible to quickly detect a pilotless flying object flying inanother directivity setting direction.

Directivity processor 63 determines whether or not the audio data whichis collected by microphone array MAk and converted to digital values byA/D converters An1 to Aq is stored temporarily in memory 38 (S22). In acase in which the sound data is not stored (NO in S22), the process ofdirectivity processor 63 returns to step S21.

When the sound data which is collected by microphone array MA istemporarily stored in memory 38 (YES in S22), directivity processor 63performs beam forming on an arbitrary directivity setting direction BF2in directivity setting area BF1 of monitoring area 8, which is otherthan the masking area set by masking area setter 69 a, and performs anextraction process on the sound data of directivity setting directionBF2 (S23).

Frequency analyzer 64 detects the frequency and sound pressure of thesound data which is subjected to the extraction process (S24).

Object detector 65 compares the detected sound pattern which isregistered in the pattern memory of memory 38 with the detected soundpattern which is obtained as a result of the frequency analysisprocessing and performs detection of pilotless flying object (S25).

Detection result determiner 66 notifies output controller 35 of thecomparison results, and notifies detecting direction controller 68 ofthe detection direction transition (S26).

For example, object detector 65 compares the detected sound patternwhich is obtained as a result of the frequency analysis processing tofour frequencies f1, f2, f3, and f4 which are registered in the patternmemory of memory 38. As a result of the comparison, in a case in whichthe both detected sound patterns include at least two of the samefrequency and the sound pressures of the frequencies are greater thanfirst threshold th1, object detector 65 determines that the patterns ofboth detected sounds are similar and that pilotless flying object do ispresent.

Here, a case is assumed in which at least two frequencies match;however, object detector 65 may determine similarity in a case in whicha single frequency matches and the sound pressure of the frequency isgreater than first threshold th1.

Object detector 65 may set an allowed frequency error in relation toeach frequency, and may determine whether or not there is similarity bytreating frequencies within the frequency error range as the samefrequency.

In addition to the comparison of frequencies and sound pressures, objectdetector 65 may perform determination by adding substantial matching ofsound pressure ratios of the sounds of the frequencies to thedetermination conditions. In this case, since the determinationconditions become stricter, it becomes easier for sound source detectionunit UDk to identify a detected pilotless flying object dn as the target(pilotless flying object dn) which is registered in advance, and it ispossible to improve the detection precision of pilotless flying objectdn.

Detection result determiner 66 determines whether or not pilotlessflying object dn is present as a result of step S26 (S27).

In a case in which pilotless flying object dn is present, detectionresult determiner 66 notifies output controller 35 of the fact thatpilotless flying object dn is present (detection result of pilotlessflying object dn) (S28).

Meanwhile, in step S27, in a case in which pilotless flying object dn isnot present (NO in S27), detection result determiner 66 instructsscanning controller 67 to transition directivity setting direction BF2of the scanning target in monitoring area 8 to the next differentdirection. Scanning controller 67 causes directivity setting directionBF2 of the scanning target in monitoring area 8 to transition to thenext different direction (S29). The notification of the detectionresults of pilotless flying object dn may be performed at once after thescanning of all directions is completed instead of at the timing atwhich the detection process of a single directivity setting direction iscompleted.

The order in which directivity setting direction BF2 is caused totransition in order in monitoring area 8 may be a spiral-shaped(cyclone-shaped) order in directivity setting area BF1 of monitoringarea 8 or the entire range of monitoring area 8, for example, totransition from an outside circumference toward an inside circumference,or to transition from an inside circumference to an outsidecircumference, as long as the area is other than the masking area set bymasking area setter 69 a.

Instead of scanning the directivity setting direction continually in asingle sweep, detecting direction controller 68 may set the position inmonitoring area 8 in advance and move directivity setting direction BF2to each position in an arbitrary order, as long as the area is otherthan the masking area set by masking area setter 69 a. Accordingly,monitoring apparatus 10 is capable of starting the detection processfrom positions at which pilotless flying object dn easily enter, forexample, and it is possible to improve the efficiency of the detectionprocess.

Scanning controller 67 determines whether or not the scanning iscompleted in all directions in monitoring area 8 (S30). In a case inwhich the scanning is not completed in all directions (NO in S30), theprocess of directivity processor 63 returns to step S23, and the sameprocesses are performed. In other words, directivity processor 63performs beam forming in directivity setting direction BF2 of theposition which is moved in step S29, and subjects the sound data ofdirectivity setting direction BF2 to an extraction process. Accordingly,since even if a single pilotless flying object dn is detected, thedetection of pilotless flying objects dn which may also be present iscontinued, sound source detection unit UDk is capable of detecting aplurality of pilotless flying objects dn.

Meanwhile, when the scanning is completed in all directions in step S30(YES in S30), directivity processor 63 erases the sound data which istemporarily stored in memory 38 and is collected by microphone array MAk(S31).

After the erasing of the sound data, signal processor 33 determineswhether or not the detection process of pilotless flying objects dn iscompleted (S32). The completion of the detection process of pilotlessflying objects dn is performed in accordance with a predetermined event.For example, in step S6, the number of times pilotless flying object dnwas not detected is held in memory 38, and in a case in which the numberof times is greater than or equal to a predetermined number, thedetection process of pilotless flying objects dn may be completed.Signal processor 33 may complete the detection process of pilotlessflying object dn based on a time expiration of a timer, or useroperation of a user interface (UI) included in console 32. The detectionprocess may be completed in a case in which the power of monitoringapparatus 10 is turned off.

In the process of step S24, frequency analyzer 64 analyses the frequencyand measures the sound pressure of the frequency. Detection resultdeterminer 66 may determine that pilotless flying object dn isapproaching sound source detection unit UD when the sound pressure levelwhich is measured by frequency analyzer 64 gradually increases with thepassage of time.

For example, in a case in which the sound pressure level of apredetermined frequency which is measured at time t11 is smaller thanthe sound pressure level of the same frequency measured at time t12,which is later than time t11, the sound pressure is increasing with thepassage of time, and pilotless flying object dn may be determined asapproaching. The sound pressure level may be measured over three or moretimes, and pilotless flying object dn may be determined as approachingbased on the transition of a statistical value (for example, a variancevalue, an average value, a maximum value, a minimum value, or the like).

In a case in which the measured sound pressure level is greater than awarning threshold, which is a warning level, detection result determiner66 may determine that pilotless flying object dn entered a warning area.

The warning threshold is a greater value than above-described thirdthreshold th3, for example. The warning area is the same area asmonitoring area 8, or is an area which is contained in monitoring area 8and is narrower than monitoring area 8, for example. The warning area isan area for which entrance by pilotless flying objects dn is restricted,for example. The approach determination and the entrance determinationof pilotless flying objects dn may be executed by detection resultdeterminer 66.

FIG. 12 is a diagram illustrating an example of a display screen offirst monitor MN1 in a case where masking area MSK3 is not set. In FIG.12, in first monitor MN1, pilotless flying object dn detected bymonitoring apparatus 10 is visible on omnidirectional image IMG1 on theupper right side of the paper of FIG. 12. In addition, the soundpressure heat map respectively corresponding to the sound sourcesgenerated by pilotless flying object dn or in the range of an imagingangle of view of omnidirectional camera CAk is superimposed anddisplayed.

As described with reference to FIG. 9, in the exemplary embodiment, ifthe sound pressure value on a per-pixel basis calculated by sound sourcedirection detector 34 is less than or equal to first threshold th1, thearea is displayed colorless, if the sound pressure value is greater thanfirst threshold th1 and is less than or equal to second threshold th2,the area is displayed in blue, if the sound pressure value is greaterthan second threshold th2 and less than or equal to third threshold th3,the area is displayed in pink, and if the sound pressure value isgreater than third threshold th3, the area is displayed in red.

In FIG. 12, for example, in the vicinity of the rotary blade or therotor around the body center of pilotless flying object dn, since thesound pressure value is greater than third threshold th3, the areas arerendered by red areas RD1, RD2, RD3, and RD4. Similarly, the areasaround the red areas are rendered by pink areas PD1, PD2, PD3, and PD4which indicate that the sound pressure value is the greatest next to thered area. Similarly, the areas around the pink areas are rendered byblue areas BD1, BD2, BD3, and BD4 which indicate that the sound pressurevalue is the greatest next to the pink area.

In addition, FIG. 12 illustrates that the sound source is also presentin office buildings, and pixels at which the sound pressure value on aper-pixel basis calculated by sound source direction detector 34 isgreater than third threshold th3 or sets of such pixels are rendered byred areas R1, R2, R3, R4, R5, R6, and R7. Similarly, the areas aroundthe red areas of the office buildings are rendered by pink areas P1, P2,P3, P4, and P5 which indicate that the sound pressure value is thegreatest next to the red area. Similarly, the areas around the pinkareas of the office buildings are rendered by blue areas B1 and B2 whichindicate that the sound pressure value is the greatest next to the pinkarea. Other areas in omnidirectional image IMG1 are rendered bycolorless area N1 since the sound pressure value is less than or equalto first threshold th1, and thus do not cause deterioration ofvisibility of omnidirectional image IMG1 which is the background.

Next, the setting of the masking area in the exemplary embodiment willbe described in detail with reference to FIGS. 13, 14, and 15. FIG. 13is an explanatory diagram illustrating an example of a display of themasking area at the time of performing an automatic learning process, intime series. FIG. 14 is a sequence diagram illustrating an example of anoperation procedure of setting the masking area in the first exemplaryembodiment. FIG. 15 is a diagram illustrating an example of a displayscreen of the first monitor in a case where the masking area is set. Thesequence illustrated in FIG. 14 is the so-called initial settingsexecuted before the operation of the sequence illustrated in FIG. 9 isstarted.

In FIG. 14, the user instructs monitoring apparatus 10 to start theautomatic learning process of the masking area by using console 32, forexample (T1). Monitoring apparatus 10 performs an image transmissionrequest in relation to omnidirectional camera CAk (T2). Omnidirectionalcamera CAk starts the imaging process corresponding to the input ofpower in accordance with the image transmission request. Omnidirectionalcamera CAk transmits the data of the omnidirectional image (for example,a still image or a video) which is obtained through imaging tomonitoring apparatus 10 via network NW (T3). Monitoring apparatus 10converts the omnidirectional image data which is transmitted fromomnidirectional camera CAk into display data such as NTSC, outputs thedisplay data to first monitor MN1, and instructs first monitor MN1 todisplay the omnidirectional image data (T4). In this manner, when thedisplay data transmitted from monitoring apparatus 10 is input to firstmonitor MN1, first monitor MN1 displays the data of omnidirectionalimage IMG1 of omnidirectional camera CAk on the screen (refer to theupper left side of the paper of FIG. 13).

In addition, monitoring apparatus 10 performs an audio transmissionrequest in relation to microphone array MAk (T5). Microphone array MAstarts the sound collection process corresponding to the input of powerin accordance with the audio transmission request. Microphone array MAkencodes the audio data of monitoring area 8 obtained through soundcollection and transmits the encoded audio data to monitoring apparatus10 via network NW (T6). In monitoring apparatus 10, sound sourcedirection detector 34 calculates, as the sound parameter, the soundpressure on a per-pixel basis using the individual pixels which form theomnidirectional image data of monitoring area 8, based on theomnidirectional image data which is captured by omnidirectional cameraCAk and the audio data which is collected by microphone array MAk.

Furthermore, masking area setter 69 a determines pixels at which thecalculated value of the sound pressure by sound source directiondetector 34 is greater than or equal to a predetermined masking areathreshold (for example, third threshold th3 described above) or sets ofsuch pixels. Masking area setter 69 a saves and registers informationindicating the determined pixels or the determined sets of such pixels,as information indicating the masking area, in memory 38 (T7).Specifically, the information indicating the masking area is coordinateson the omnidirectional image which specify the position of a pixel atwhich the calculated value of the sound pressure is greater than orequal to the masking area threshold. Masking area setter 69 a outputs,to first monitor MN1 via output controller 35, the informationindicating the masking area and an instruction of causing the maskingarea (pixels at which the calculated value of the sound pressure isgreater than or equal to the masking area threshold, or sets of suchpixels) to be filled with a predetermined color (for example, red) (T8).In this manner, first monitor MN1 performs a process of filling theposition of coordinates corresponding to masking area MSK1 onomnidirectional image IMG, with a predetermined color through theinstruction transmitted from monitoring apparatus 10 (refer to the upperright side of the paper of FIG. 13). In omnidirectional image IMG1 onthe upper right side of the paper of FIG. 13, masking area MSK1indicates the entire area which is filled with a predetermined color.

Similarly, microphone array MA encodes the audio data of monitoring area8 obtained through sound collection which is continuously beingperformed, and transmits the encoded audio data to monitoring apparatus10 via network NW in accordance with the audio transmission request frommonitoring apparatus 10 (T9). In monitoring apparatus 10, sound sourcedirection detector 34 calculates, as the sound parameter, the soundpressure on a per-pixel basis using the individual pixels which form theomnidirectional image data of monitoring area 8, based on theomnidirectional image data which is captured by omnidirectional cameraCAk and the audio data which is collected by microphone array MAk.

Furthermore, masking area setter 69 a determines pixels at which thecalculated value of the sound pressure by sound source directiondetector 34 is greater than or equal to the masking area threshold orsets of such pixels. Masking area setter 69 a saves and registersinformation indicating the determined pixels or the determined sets ofsuch pixels, as information indicating the masking area, in memory 38(T10). Masking area setter 69 a outputs, to first monitor MN1 via outputcontroller 35, the information indicating the masking area and theinstruction of causing the masking area to be filled with apredetermined color (for example, red) (T11). In this manner, firstmonitor MN1 performs a process of filling the position of coordinatescorresponding to masking area MSK2, which is accumulated on masking areaMSK1, on omnidirectional image IMG, with a predetermined color throughthe instruction transmitted from monitoring apparatus 10 (refer to thelower right side of the paper of FIG. 13). In omnidirectional image IMG1on the lower right side of the paper of FIG. 13, masking area MSK2indicates the entire area which is filled with a predetermined color.

Here, the completion of the automatic learning process of the maskingarea is instructed to monitoring apparatus 10 through the user'soperation using console 32 (T12). In accordance with the instruction,monitoring apparatus 10 transmits an audio transmission suspensionrequest to microphone array MAk (T13). In this manner, microphone arrayMAk suspends the distribution (transmission) of the audio data ofmonitoring area 8, which is obtained through the sound collection, tomonitoring apparatus 10.

In addition, if an operation of correcting the masking area (that is,addition or deletion of the masking area) is performed on first monitorMN1 on which masking area MSK2 of the lower right side of the paper ofFIG. 13 is illustrated, through the user's operation using console 32(T14), in monitoring apparatus 10, masking area setter 69 a, through theuser's operation, adds the designated position on omnidirectional imageIMG1 as the masking area or deletes the designated position from themasking area, and then saves and registers information indicatingmasking area MSK3 after the correction, in memory 38 (T15).

In step T14, for example, an operation (for example, range designationoperation) for deleting areas GOM1, GOM2, and GOM3, which are determinedto be unnecessary as the masking area by the user, or an operation (forexample, rendering operation) for adding the entire area of officebuildings of the background covered by masking area MSK2 as the maskingarea is performed on first monitor MN1 on which masking area MSK2 of thelower right side of the paper of FIG. 13 is illustrated.

Masking area setter 69 a outputs, to first monitor MN1 via outputcontroller 35, the information indicating the masking area and theinstruction of causing the masking area to be filled with apredetermined color (for example, red) (T16). In this manner, firstmonitor MN1 performs a process of filling the position of coordinatescorresponding to masking area MSK3, which is accumulated on maskingareas MSK1 and MSK2, on omnidirectional image IMG, with a predeterminedcolor through the instruction transmitted from monitoring apparatus 10(refer to the lower left side of the paper of FIG. 13). Inomnidirectional image IMG1 on the lower left side of the paper of FIG.13, masking area MSK3 indicates the entire area which is filled with apredetermined color. Accordingly, the sky is excluded from the settingof masking area MSK3.

According to the sequence illustrated in FIG. 14, in view of a tendencyin which pilotless flying object dn as the detection target of theexemplary embodiment flies in the sky around which the sound sources arerarely present in the range of the imaging angle of view ofomnidirectional camera CAk, since no sound source in which the soundpressure greater than or equal to the masking area threshold isgenerated, is found in the sky according to the analysis of the audiodata from microphone array MAk, it is possible to set the sky as thedetection target of pilotless flying object dn. Meanwhile, by settingthe entire area of office buildings around which plural sound sourcesmay present, as the masking area, it is possible to prevent the soundsource other than pilotless flying object dn from being erroneouslydetected as pilotless flying object dn which is originally desired to bedetected, and it is possible to improve the detection precision and thespeed of the detection process for pilotless flying object dn.

In other words, as illustrated in FIG. 15, in omnidirectional imageIMG1, detection of pilotless flying object dn is performed only in otherareas except for masking area MSK3. As a result, in comparison withomnidirectional image IMG1 illustrated in FIG. 12, a sound pressure heatmap in which a sound pressure value of the sound generated at the soundsource position is converted into the visual image is superimposed anddisplayed on the sound source position around pilotless flying object dndetected in other areas except for masking area MSK3. Meanwhile,superimposition and display of the sound pressure heat map in which asound pressure value of the sound generated at the sound source positionis converted into the visual image, is omitted for areas around othersound sources (f_(o)r example, angry voice of a person in officebuildings) which are detected in masking area MSK3 but are not thepilotless flying object.

In the exemplary embodiment, in a case where the masking area is set bymasking area setter 69 a, as illustrated in FIG. 15, the sound pressureheat map in which a sound pressure value of the sound generated at thesound source position is converted into the visual image, issuperimposed and displayed around pilotless flying object dn. However,in a case where the masking area is not set by masking area setter 69 a,as illustrated in FIG. 12, the sound pressure heat map corresponding tothe sound pressure value of the detected sound source may besuperimposed and displayed on omnidirectional image IMG1.

As described above, in pilotless flying object detection system 5according to the exemplary embodiment, monitoring apparatus 10 sets, bymasking area setter 69 a, a masking area to be excluded from thedetection of pilotless flying object dn, which appears in the capturedimage (omnidirectional image IMG1) of monitoring area 8 by using theaudio data collected by microphone array MAk. Monitoring apparatus 10detects pilotless flying object dn in other areas except for the maskingarea by using the audio data collected by microphone array MAk andinformation indicating the masking area. In addition, in a case wherepilotless flying object dn is detected in an area other than the maskingarea, monitoring apparatus 10 superimposes a sound source visual image(that is, visual images of red area RD1, pink area PD1, blue area BD1,and the like) which indicate the volume of the sound of the sound sourceposition, at the sound source position of pilotless flying object dn inomnidirectional image IMG1 and displays the result on first monitor MN1.

In this manner, since in pilotless flying object detection system 5, itis possible to automatically set a masking area to be excluded from thedetection process of pilotless flying object dn as the detection target,with respect to monitoring area 8 as the imaging target ofomnidirectional camera CAk, it is possible to reduce the possibility oferroneously detecting an object at the sound source position in themasking area as pilotless flying object dn, and to suppressdeterioration of the detection precision of pilotless flying object dn.In addition, in pilotless flying object detection system 5, it ispreferable that the detection determination of pilotless flying objectdn is performed only for the areas except for the masking area, withoutthe necessity of detecting pilotless flying object dn over the imagingangle of view (that is, entire region of omnidirectional image IMG1) ofomnidirectional camera CAk, and thus it is possible to further enhancethe detection process of pilotless flying object dn.

In addition, in pilotless flying object detection system 5, sound sourcedirection detector 34 calculates the sound pressure specifying thevolume of the sound of monitoring area 8 on a per-predetermined-unitbasis of pixels, which form omnidirectional image IMG1, based on theaudio data collected by microphone array MAk. Masking area setter 69 asuperimposes and displays a position of the sound source in which thecalculated value of the sound pressure is greater than or equal to themasking area threshold relating to the volume of the sound, or an areaincluding the position, on first monitor MN1, and further, sets thesound source area displayed on first monitor MN1 as the masking areathrough the user's confirmation operation. In this manner, it ispossible for the user to easily set a place where the possibility of theflying of pilotless flying object do is low but other sound sources (forexample, angry voice of a person) may be generated, as the masking areato be excluded from areas of the detection target of pilotless flyingobject do while visually checking first monitor MN1.

In addition, masking area setter 69 a sets the sound source area afterthe user's adding operation as the masking area through the user'sadding operation for further adding the sound source area (that is,areas as candidates for the masking area) displayed on first monitorMN1. In this manner, it is possible for the user to set the masking areaby easily designating a location that the user desires to add as themasking area under the user's determination, while visually checking thelocation, which is automatically filled with a predetermined color asthe candidate for the masking area by monitoring apparatus 10, on firstmonitor MN1, and thus the usability of the user is improved.

In addition, masking area setter 69 a sets the sound source area afterthe user's deleting operation as the masking area through the user'sdeleting operation for deleting at least a part of the sound source area(that is, areas as candidates for the masking area) displayed on firstmonitor MN1. In this manner, it is possible for the user to set themasking area by easily designating a part of a location that the userdesires to exclude from the location filled with a predetermined coloras the masking area, under the user's determination, while visuallychecking the location, which is automatically filled with apredetermined color as the candidate for the masking area by monitoringapparatus 10, on first monitor MN1, and thus the usability of the useris improved.

In addition, output controller 35 superimposes the sound source visualimage, in which the sound pressure is converted into a different visualimage in stages according to comparison between the calculated value ofthe sound pressure and a plurality of thresholds relating to the volumeof the sound, on a per-predetermined-unit basis of pixels which formomnidirectional image IMG1 of monitoring area 8 and displays the resulton first monitor MN1. In this manner, by viewing first monitor MN1, itis possible for the user to not only acknowledge a broad range of asituation of monitoring area 8 as the omnidirectional image, but alsoeasily check the place of the generation source (for example, pilotlessflying object dn) of the sound generated in an area other than themasking area of monitoring area 8 and the volume of the sound, as thevisual image, in omnidirectional image IMG1 of monitoring area 8captured by omnidirectional camera CAk.

History Leading to Second Exemplary Embodiment

Japanese Patent Unexamined Publication No. 2006-168421 described abovediscloses that a sound pressure in a frequency unique to the flyingobject such as a helicopter or Cessna is compared with a predeterminedset level, and if the sound pressure is greater than or equal to the setlevel, it is determined to be the flying object as the monitoringtarget.

However, in Japanese Patent Unexamined Publication No. 2006-168421described above, it is not considered to quantitatively illustrate thelevel to which the measured sound pressure corresponds, of the soundpressure in which a plurality of levels are prescribed. Thus, there is aproblem in that when any sound is detected in the imaging area of thecamera device, it is difficult to specifically present the volume of thesound as detailed visual information for sound, regardless of themagnitude of the volume of the detected sound at the sound sourceposition.

Therefore, a second exemplary embodiment describes an example of amonitoring system which presents in detail, in stages, the volume of thedetected sound at the sound source position in the imaging area of thecamera device, regardless of the magnitude of the volume of the sound atthe sound source position, and assists the user in accuratelyascertaining the volume of the sound at the sound source position.

Second Exemplary Embodiment

In the second exemplary embodiment, since the internal configuration ofeach device configuring pilotless flying object detection system 5 isthe same as the internal configuration of each device configuringpilotless flying object detection system 5 according to the firstexemplary embodiment, the same reference numeral is assigned to the samecontents and the description of the same contents is not repeated, anddifferent contents will be described.

In the second exemplary embodiment, after generating and displaying thesound pressure heat map on first monitor MN1 described in the firstexemplary embodiment, monitoring apparatus 10 analyzes in detail, thesound pressure heat map according to the relationship between thecalculated value of the sound pressure which is required for generatingthe sound pressure heat map and a plurality of thresholds (refer tolater description) and displays the result, through the user's operationwith respect to console 32 (refer to later description). Hereinafter,three analysis methods will be described.

First Analysis Method

In the first analysis method, after monitoring apparatus 10 superimposesthe sound pressure heat map corresponding to omnidirectional image IMG2,onto omnidirectional image IMG2 and displays the result on first monitorMN1, if the user designates a partial range of omnidirectional imageIMG2, monitoring apparatus 10 changes the display resolution of thesound pressure heat map of the designated range to be the same as thedisplay resolution of omnidirectional image IMG2 which is the entireimage. The operation example of the first analysis method will bedescribed with reference to FIGS. 16 and 17. FIG. 16 is an explanatorydiagram illustrating an outline of a dynamic change of the displayresolution of the sound pressure heat map in the second exemplaryembodiment. FIG. 17 is a flowchart illustrating an example of anoperation procedure of the dynamic change of the display resolution ofthe sound pressure heat map in the second exemplary embodiment.

In FIG. 17, monitoring apparatus 10 receives the audio data ofmonitoring area 8 transmitted from microphone array MAk, as an input(S41). It is determined whether or not a partial cut-out range ofomnidirectional image IMG2 (refer to the upper left side of the paper ofFIG. 16) is designated with respect to first monitor MN1 through theuser's operation using console 32 (S42).

Here, in the upper left side of the paper of FIG. 16, omnidirectionalimage IMG2 displayed on first monitor MN1 has a display resolution of“Wmax×Hmax” in a case where the display size of omnidirectional imageIMG2 in an X direction is “Wmax” and the display size in a Y directionis “Hmax”. In addition, the coordinates of endpoints of the partialcut-out range of omnidirectional image IMG2, which is designated by theuser's operation, are (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) to forma rectangular shape. FIG. 16 illustrates only (X1, Y1) and (X2, Y2)which are present on the diagonal of the rectangular shape.

In a case where a rectangular shape of which the endpoints are fourpoints of (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) is not designatedby the user's operation (NO in S42), in order to generate a soundpressure map for the entirety of omnidirectional image IMG2, soundsource direction detector 34 of monitoring apparatus 10 sets (X, Y)=(0,0) (S43), and calculates sound pressure P(X, Y) at the coordinates of(X, Y)=(0, 0) (S45).

In a case where the X coordinate of omnidirectional image IMG2 does notmatch maximum value Wmax (NO in S46), in order to generate a soundpressure map for the entirety of omnidirectional image IMG2, soundsource direction detector 34 increases the X coordinate by one (S47),and calculates sound pressure P(X, Y) for the coordinates (X, Y) afterthe increase.

In a case where the X coordinate of omnidirectional image IMG2 matchesmaximum value Wmax (YES in S46), in a case where the Y coordinate ofomnidirectional image IMG2 does not match maximum value Hmax (NO inS48), in order to generate a sound pressure map for the entirety ofomnidirectional image IMG2, sound source direction detector 34 causesthe X coordinate to return to 0, and increases the Y coordinate by one(S49), and calculates sound pressure P(X, Y) for the coordinates (X, Y)after the increase. Sound source direction detector 34 repeats eachprocess of steps S45 to S49 until the Y coordinate of omnidirectionalimage IMG2 matches maximum value Hmax, and thereby, can generate a soundpressure map for the entirety of omnidirectional image IMG2, and savesand registers the sound pressure map in memory 38 (S50), similar to thefirst exemplary embodiment.

Meanwhile, in a case where a rectangular shape of which the endpointsare four points of (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) isdesignated by the user's operation (YES in S42), in order to generate asound pressure map for the range designated by the user's operationamong omnidirectional image IMG2, sound source direction detector 34 ofmonitoring apparatus 10 sets (X, Y)=(X1, Y1) (S44), and calculates soundpressure P(X, Y) for the coordinates of (X, Y)=(X1, Y1) (S45).

Furthermore, in a case where the X coordinate of omnidirectional imageIMG2 does not match maximum value Wmax (NO in S46), in order to generatea sound pressure map for the range designated by the user's operationamong omnidirectional image IMG2, sound source direction detector 34increases the X coordinate by (X2−X1)/Wmax (S47), and calculates soundpressure P(X, Y) for the coordinates (X, Y) after the increase.

In a case where the X coordinate of omnidirectional image IMG2 matchesmaximum value Wmax (YES in S46), in a case where the Y coordinate ofomnidirectional image IMG2 does not match maximum value Hmax (NO inS48), in order to generate a sound pressure map for the range designatedby the user's operation among omnidirectional image IMG2, sound sourcedirection detector 34 causes the X coordinate to return to X1, andincreases the Y coordinate by (Y2−Y1)/Hmax (S49), and calculates soundpressure P(X, Y) for the coordinates (X, Y) after the increase. Soundsource direction detector 34 repeats each process of steps S45 to S49until the Y coordinate of omnidirectional image IMG2 matches maximumvalue Hmax, and thereby, can generate a sound pressure map for the rangedesignated by the user's operation among omnidirectional image IMG2, andsaves and registers the sound pressure map in memory 38 (S50). Afterstep S50, the process of monitoring apparatus 10 returns to step S41,and repeats the processes of steps S42 to S50 for the audio data whichis input in step S41.

Accordingly, as illustrated in the upper right side of the paper of FIG.16, if monitoring apparatus 10 simply cuts out the range designated bythe user's operation, generates an audio heat map according to the soundpressure value on a per-pixel basis using pixels which form the cut-outrange, and superimposes the result on first monitor MN1, only an imagewith a low (that is, rough) display resolution is displayed.

However, by the first analysis method of the exemplary embodiment,monitoring apparatus 10 calculates sound pressure P(X, Y) for thepartial cut-out range of omnidirectional image IMG2 designated by theuser's operation, on a per-fine unit basis (that is, for every(X2−X1)/Wmax in the X direction, and for every (Y2−Y1)/Hmax in the Ydirection) such that the display resolution of the cut-out range is thesame as the display resolution of the entirety of omnidirectional imageIMG2. In this manner, as illustrated in the lower right side of thepaper of FIG. 16, monitoring apparatus 10 can accurately display thesound pressure heat map of the cut-out range designated by the user'soperation on first monitor MN1 with a resolution (unit) finer than thedisplay resolution of the sound pressure heat map when simply cuttingout the range, and thus can cause the user to accurately ascertain thedetails of the distribution of the sound sources in the cut-out rangedesignated by the user's operation.

Second Analysis Method

In the exemplary embodiment, before the description of the secondanalysis method, as maters in common in FIGS. 18 to 20, a plurality ofinter-threshold widths which define the volume of the sound in stageswill be described. FIG. 18 is an explanatory diagram illustrating anoutline of width adjustment of thresholds according to a frequencydistribution of sound pressure values, and a display result of acaptured image accompanied by the width adjustment in the secondexemplary embodiment. In FIGS. 18 to 20, it is defined that as the soundsource visual images corresponding to the inter-thresholds, anultramarine image, an indigo image, a blue image, a sky blue image, abluish green image, a yellowish green image, a yellow image, an orangeimage, a red image, and a crimson image are used as the sound pressurevalue is toward the upper limit from the lower limit. In a case wherethe sound pressure value is the minimum (that is, lower limit), theultramarine image is used, and in a case where the sound pressure valueis the maximum (that is, upper limit), the crimson image is used.

For example, as illustrated in the left side of the paper of FIG. 18,ten thresholds are defined in total so as to correspond to the volume ofthe sound pressure value, and in the example on left side of the paperof FIG. 18, scales provided to the axis of each sound pressure valuecorrespond to thresholds, respectively. Therefore, in the example on theleft side of the paper of FIG. 18, in a case where the sound pressurevalue is included between a certain threshold (for example, 5) and athreshold (for example, 6) that is great next to the certain threshold,a color image (for example, the yellowish green image) corresponding tothe inter-threshold in which the sound pressure value is included isused as the sound source visual image for visually indicating the soundpressure value when the sound pressure heat map is generated.

In the second analysis method, threshold adjuster 69 b of monitoringapparatus 10 dynamically changes ten thresholds in total orinter-threshold widths thereof, based on the frequency of generation (inother words, the frequency distribution) of the sound pressure value,which is calculated on the per-pixel basis (also possible on theper-predetermined-unit basis, the same applies to the following) usingpixels which form the omnidirectional image when output controller 35generates the sound pressure heat map corresponding to theomnidirectional image. That is, threshold adjuster 69 b dynamicallychanges the setting of the correspondence relationship between aplurality of thresholds and the sound source visual images according tothe omnidirectional image. For example, with reference to FIG. 18,threshold adjuster 69 b holds the setting in which the yellowish greenimage is used as the sound source visual image if the sound pressurevalue is between threshold 5 and threshold 6. However, if the frequencyof appearance of the sound pressure value between threshold 5 andthreshold 6 is high among pixels of the entirety of the omnidirectionalimage, threshold adjuster 69 b changes inter-threshold width LG1 (forexample, between threshold 5 and threshold 6) for using the yellowishgreen image to width LG2 (for example, between threshold 4.5 andthreshold 4.8) which is narrower than width LG1.

Accordingly, in a case where the sound pressure values on the per-pixelbasis of pixels which form the omnidirectional image are concentrated oninter-threshold width AR1 for using an image with a specific color, asillustrated in the lower left side of the paper of FIG. 18, as soundpressure heat map VMP1, images with the same color or images withvisually similar colors (for example, the yellowish green image and thebluish green image) are used as the sound source visual images, and thusit is difficult to present in detail, to a user, the distribution of thesound sources appearing in the omnidirectional image.

However, according to the second analysis method of the exemplaryembodiment, monitoring apparatus 10 dynamically changes the setting ofthe correspondence relationship between the sound source visual imagesand a plurality of thresholds according to the frequency of appearance(frequency distribution) of the sound pressure value on a per-pixelbasis using the pixels which form the omnidirectional image by thresholdadjuster 69 b, reflects the change, and then displays sound pressureheat map VMP2 corresponding to the omnidirectional image on firstmonitor MN1. In this manner, as illustrated in the lower right side ofthe paper of FIG. 18, monitoring apparatus 10 can present to a user, adetailed distribution of the sound pressures as sound pressure heat mapVMP2, and thus can cause the user to accurately ascertain thedistribution of the sound source position.

Third Analysis Method

In the third analysis method, monitoring apparatus 10 can arbitrarilydesignate the upper limit, the lower limit or both the limits of thethresholds, which are for defining the use of the sound source visualimage (that is, color image), through the user's operation using console32. FIGS. 19A and 19B are explanatory diagrams illustrating an outlineof a setting change of inter-threshold widths defining the use of thesound source visual image in the second exemplary embodiment.

For example, in FIG. 19A, the inter-threshold (upper end threshold)width for defining the use of the sound source visual image (that is,the crimson image) indicating that the sound pressure value is thehighest (that is, the upper limit) is changed from between threshold 9and threshold 10, to between threshold 6 and threshold 10, and further,the inter-threshold (lower end threshold) width for defining the use ofthe sound source visual image (that is, the ultramarine image)indicating that the sound pressure value is the lowest (that is, thelower limit) is changed from between threshold 0 and threshold 1, tobetween threshold 0 and threshold 2. In this manner, monitoringapparatus 10 can change the inter-threshold widths such that the widthsbetween the remaining eight thresholds are different from the twointer-threshold widths changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining eight thresholdsto be equal intervals.

In FIG. 19B, after the widths are changed as illustrated in FIG. 19A,the inter-threshold (upper end threshold) width for defining the use ofthe sound source visual image (that is, the crimson image) indicatingthat the sound pressure value is the highest (that is, the upper limit)is changed from between threshold 6 and threshold 10, to betweenthreshold 9.3 and threshold 10, and further, the inter-threshold (lowerend threshold) width for defining the use of the sound source visualimage (that is, the ultramarine image) indicating that the soundpressure value is the lowest (that is, the lower limit) is changed frombetween threshold 0 and threshold 2, to between threshold 0 andthreshold 5. In this manner, similarly, monitoring apparatus 10 canchange the inter-threshold widths such that, though each threshold isdifferent from the example illustrated in FIG. 19A, the widths betweenthe remaining eight thresholds are different from the twointer-threshold widths changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining eight thresholdsto be equal intervals.

In FIGS. 19A and 19B, examples are described in which both theinter-threshold (upper end threshold) width for defining the use of thesound source visual image (that is, the crimson image) indicating thatthe sound pressure value is the highest (that is, the upper limit) andthe inter-threshold (lower end threshold) width for defining the use ofthe sound source visual image (that is, the ultramarine image)indicating that the sound pressure value is the lowest (that is, thelower limit) are changed, but the same applies to a case in which onlyone is changed.

That is, in a case where only the inter-threshold(upper end threshold)width for defining the use of the sound source visual image (that is,the crimson image) indicating that the sound pressure value is thehighest (that is, the upper limit) is changed, similarly, monitoringapparatus 10 can change the inter-threshold widths such that the widthsbetween the remaining nine thresholds are different from the oneinter-threshold width changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining nine thresholds tobe equal intervals.

In addition, in a case where only the inter-threshold (lower endthreshold) width for defining the use of the sound source visual image(that is, the ultramarine image) indicating that the sound pressurevalue is the lowest (that is, the lower limit) is changed, similarly,monitoring apparatus 10 can change the inter-threshold widths such thatthe widths between the remaining nine thresholds are different from theone inter-threshold width changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining nine thresholds tobe equal intervals.

An operation relating to the setting change of the inter-threshold widthaccording to the third analysis method will be described with referenceto FIGS. 20 and 21. FIG. 20 is an explanatory diagram illustrating anoutline of a display of the captured image accompanied by the settingchange of the inter-threshold widths defining the use of the crimsonimage and the ultramarine image in the second exemplary embodiment. FIG.21 is a flowchart illustrating an example of an operation procedure ofthe setting change of inter-threshold widths in the second exemplaryembodiment.

As illustrated in FIG. 20, it is assumed that both the inter-thresholdwidths for respectively defining the use of the sound source visualimage (that is, the crimson image) indicating that the sound pressurevalue is the highest (that is, the upper limit) and the use of the soundsource visual image (that is, the ultramarine image) indicating that thesound pressure value is the lowest (that is, the lower limit) arechanged. In this case, as illustrated in FIG. 21, in monitoringapparatus 10, threshold adjuster 69 b dynamically changes the setting ofa correspondence table (not illustrated, correspondence relationship)between the sound source visual images and the thresholds or theinter-threshold widths defining the use of the sound source visualimage.

Accordingly, though in the example illustrated in FIG. 20, soundpressure heat map VMP2 superimposed onto the omnidirectional image (thatis, the captured image) is configured of only the crimson image,monitoring apparatus 10 generates sound pressure heat map VMP2A usingthe sound source visual image with kinds of fine colors, through theuser's operation, and superimposes sound pressure heat map VMP2A ontothe omnidirectional image (that is, the captured image) and displays theresult on first monitor MN1. The user's operation refers to, forexample, an operation of inputting a threshold or an inter-thresholdwidth displayed on first monitor MN1, with respect to an input screen(not illustrated), or an operation of dragging the inter-threshold widthdisplayed on first monitor MN1, to the display screen, but is notlimited thereto.

In FIG. 21, threshold adjuster 69 b determines whether or not theinter-threshold width defining the use of the sound source visual image(that is, the crimson image) indicating that the sound pressure value isthe highest (that is, the upper limit) is changed through the user'soperation using console 32 (S61). In a case where the inter-thresholdwidth defining the use of the sound source visual image (that is, thecrimson image) indicating that the sound pressure value is the highest(that is, the upper limit) is changed (YES in S61), threshold adjuster69 b determines whether or not the inter-threshold width defining theuse of the sound source visual image (that is, the ultramarine image)indicating that the sound pressure value is the lowest (that is, thelower limit) is changed (S62).

In a case where the inter-threshold width defining the use of the soundsource visual image (that is, the ultramarine image) indicating that thesound pressure value is the lowest (that is, the lower limit) is changed(YES in S62), threshold adjuster 69 b corrects the correspondence tablebetween the sound source visual images and the thresholds or theinter-threshold widths defining the use of the sound source visualimage, according to the change result (S63). For example, thresholdadjuster 69 b changes the inter-threshold widths such that the widthsbetween the remaining eight thresholds are different from the twointer-threshold widths changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining eight thresholdsto be equal intervals. In this manner, for example, in a case whereplural sound pressure values between the threshold defining the use ofthe crimson image and the threshold defining the use of the ultramarineimage are obtained, monitoring apparatus 10 can display in detail, thedistribution around pixels at which plural sound pressure values areconcentrated, as the sound pressure heat map through the user'soperation of adjusting the inter-threshold width.

In a case where the inter-threshold width defining the use of the soundsource visual image (that is, the ultramarine image) indicating that thesound pressure value is the lowest (that is, the lower limit) is notchanged (NO in S62), threshold adjuster 69 b corrects the correspondencetable between the sound source visual images and the thresholds or theinter-threshold widths defining the use of the sound source visualimage, according to the change result (S64). For example, thresholdadjuster 69 b changes the inter-threshold widths such that the widthsbetween the remaining nine thresholds are different from the oneinter-threshold width changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining nine thresholds tobe equal intervals. In this manner, for example, in a case where pluralsound pressure values lower than or equal to the threshold defining theuse of the crimson image are obtained, monitoring apparatus 10 candisplay in detail, the distribution around pixels at which plural soundpressure values are concentrated, as the sound pressure heat map throughthe user's operation of adjusting the inter-threshold width.

Meanwhile, in step S61, in a case where the inter-threshold widthdefining the use of the sound source visual image (that is, the crimsonimage) indicating that the sound pressure value is the highest (that is,the upper limit) is not changed (NO in S61), threshold adjuster 69 bdetermines whether or not the inter-threshold width defining the use ofthe sound source visual image (that is, the ultramarine image)indicating that the sound pressure value is the lowest (that is, thelower limit) is changed (S65). In a case where the inter-threshold widthdefining the use of the sound source visual image (that is, theultramarine image) indicating that the sound pressure value is thelowest (that is, the lower limit) is not changed (NO in S65), theprocess of threshold adjuster 69 b returns to step S61.

In a case where the inter-threshold width defining the use of the soundsource visual image (that is, the ultramarine image) indicating that thesound pressure value is the lowest (that is, the lower limit) is changed(YES in S65), threshold adjuster 69 b corrects the correspondence tablebetween the sound source visual images and the thresholds or theinter-threshold widths defining the use of the sound source visualimage, according to the change result (S66). For example, thresholdadjuster 69 b changes the inter-threshold widths such that the widthsbetween the remaining nine thresholds are different from the oneinter-threshold width changed by the user's operation by dynamicallychanging the inter-threshold widths for the remaining nine thresholds tobe equal intervals. In this manner, for example, in a case where pluralsound pressure values equal to or greater than the threshold definingthe use of the ultramarine image are obtained, monitoring apparatus 10can display in detail, the distribution around pixels at which pluralsound pressure values are concentrated, as the sound pressure heat mapthrough the user's operation of adjusting the inter-threshold width.

As described above, in pilotless flying object detection system 5 of theexemplary embodiment, monitoring apparatus 10 calculates the soundpressure specifying the volume of the sound of monitoring area 8 on aper-predetermined-unit basis of pixels, which form the captured image(omnidirectional image IMG2) of monitoring area 8, by using the audiodata collected by microphone array MAk. Monitoring apparatus 10superimposes the sound source visual image, in which the sound pressureis converted in stages into a different visual image according tocomparison between the calculated value of the sound pressure and aplurality of thresholds relating to the volume of the sound, on aper-predetermined-unit basis of pixels which form the captured image anddisplays the result on first monitor MN1. When any sound source positionis designated in the captured image on which the sound source visualimage is superimposed, monitoring apparatus 10 calculates a soundpressure for each value obtained by dividing a predetermined unit ofpixels which form a rectangular range including the sound sourceposition by a ratio between sizes of the captured image and therectangular range.

In this manner, monitoring apparatus 10 can accurately display the soundpressure heat map of the rectangular range (cut-out range) designated bythe user's operation on first monitor MN1 with a resolution (unit) finerthan the display resolution of the sound pressure heat map when simplycutting out the range, and thus can cause the user to accuratelyascertain the details of the distribution of the sound sources in thecut-out range designated by the user's operation. In other words, inmonitoring apparatus 10, it is possible to present in detail, in stages,the volume of the detected sound at the sound source position inmonitoring area 8 of omnidirectional camera CAk, regardless of themagnitude of the volume of the sound at the sound source position, andto assist the user in accurately ascertaining the volume of the sound atthe sound source position.

In addition, in pilotless flying object detection system 5 of theexemplary embodiment, monitoring apparatus 10 calculates the soundpressure specifying the volume of the sound of monitoring area 8 on aper-predetermined-unit basis of pixels, which form the captured image(omnidirectional image IMG2) of monitoring area 8, by using the audiodata collected by microphone array MAk. Monitoring apparatus 10dynamically changes the setting of a correspondence relationship betweeneach threshold of a plurality of thresholds defining the volume of thesound in stages and the sound source visual image in which the soundpressure is converted in stages into a different visual image accordingto comparison between the sound pressure and each threshold, accordingto the captured image (that is, omnidirectional image) of monitoringarea 8. Monitoring apparatus 10 superimposes, onto the captured image,the sound source visual image corresponding to the calculated value ofthe sound pressure, on the per-predetermined-unit basis of pixels whichform the captured image, based on the calculated value of the soundpressure and the changed setting of the correspondence relationship, anddisplays the result on first monitor MN1.

In this manner, monitoring apparatus 10 can present to a user, adetailed distribution of the sound pressures as the sound pressure heatmap for visually indicating the position of the sound source collectedin monitoring area 8, according to the captured image captured byomnidirectional camera CAk, and thus can cause the user to accuratelyascertain the distribution of the sound source position.

In addition, threshold adjuster 69 b of monitoring apparatus 10 changesthe inter-threshold width defining the sound source visual image basedon the frequency of appearance of the sound pressure on the per-pixelbasis using the pixels which form the captured image of monitoring area8. In this manner, monitoring apparatus 10 generates the sound pressureheat map after increasing kinds of the sound source visual image forpixels corresponding to the calculated value of the sound pressure ofwhich the frequency of appearance is high, and decreasing kinds of thesound source visual image for pixels corresponding to the calculatedvalue of the sound pressure of which the frequency of appearance is low.Therefore, it is possible to present a distribution of the soundpressure with fine and various color tones rather than a simple colortone, and to cause the user to accurately ascertain the distribution ofthe sound source position.

In addition, threshold adjuster 69 b of monitoring apparatus 10 changesall other inter-threshold widths to be equal intervals, except for theinter-threshold width that is changed according to the operation ofchanging the inter-threshold width defining the use of the sound sourcevisual image (that is, the crimson image) corresponding to the upperlimit value of the sound pressure. In this manner, for example, in acase where plural sound pressure values lower than or equal to thethreshold defining the use of the crimson image are obtained, monitoringapparatus 10 can display in detail, the distribution around pixels atwhich plural sound pressure values are concentrated, as the soundpressure heat map through the user's operation of adjusting theinter-threshold width.

In addition, threshold adjuster 69 b of monitoring apparatus 10 changesall other inter-threshold widths to be equal intervals, except for theinter-threshold width that is changed according to the operation ofchanging the inter-threshold width defining the use of the sound sourcevisual image (that is, the ultramarine image) corresponding to the lowerlimit value of the sound pressure. In this manner, for example, in acase where plural sound pressure values equal to or greater than thethreshold defining the use of the ultramarine image are obtained,monitoring apparatus 10 can display in detail, the distribution aroundpixels at which plural sound pressure values are concentrated, as thesound pressure heat map through the user's operation of adjusting theinter-threshold width.

In addition, threshold adjuster 69 b of monitoring apparatus 10 changesall other inter-threshold widths to be equal intervals, except for theinter-threshold widths that are changed according to the operation ofchanging the inter-threshold width defining the use of the sound sourcevisual image (that is, the crimson image) corresponding to the upperlimit value of the sound pressure and the inter-threshold width definingthe use of the sound source visual image (that is, the ultramarineimage) corresponding to the lower limit value of the sound pressure. Inthis manner, for example, in a case where plural sound pressure valuesbetween the threshold defining the use of the crimson image and thethreshold defining the use of the ultramarine image are obtained,monitoring apparatus 10 can display in detail, the distribution aroundpixels at which plural sound pressure values are concentrated, as thesound pressure heat map through the user's operation of adjusting theinter-threshold width.

History Leading to Third Exemplary Embodiment

Japanese Patent Unexamined Publication No. 2006-168421 described abovediscloses that a monitoring camera which is capable of changing animaging direction in arbitrary direction in a monitoring area isprovided, and the imaging direction of the monitoring camera is changedif a flying object such as a helicopter or Cessna is detected. In otherwords, a technique of changing the imaging direction of the monitoringcamera in order to perform imaging by focusing on the detected flyingobject is disclosed.

However, in Japanese Patent Unexamined Publication No. 2006-168421described above, a technique of displaying the captured image of theperiphery including the pilotless flying object detected in the range ofthe angle of view of the camera device with respect to the imaging areain a wide range is not considered. Therefore, there is a problem that itis difficult to visually present, to a user, the location in the imagingarea of the camera device where the pilotless flying object is detected,and what kinds of sound source are present at which locations in thesame imaging area.

In a third exemplary embodiment, an example of a monitoring system isdescribed which visually presents to a user, the location in the imagingarea of the camera device where the pilotless flying object is detected,and what kinds of sound source are present at which locations in thesame imaging area without deterioration of the visibility of thecaptured image of the camera device.

Third Exemplary Embodiment

In the third exemplary embodiment, since the internal configuration ofeach device configuring pilotless flying object detection system 5 isthe same as the internal configuration of each device configuringpilotless flying object detection system 5 according to the firstexemplary embodiment, the same reference numeral is assigned to the samecontents and the description of the same contents is not repeated, anddifferent contents will be described.

In the third exemplary embodiment, monitoring apparatus 10 generates atranslucent sound pressure heat map as a translucent image (translucentmap) of a sound pressure heat map after generating the sound pressureheat map (sound parameter map) described in the first exemplaryembodiment, and superimposes the translucent sound pressure heat maponto the omnidirectional image to display the result on first monitorMN1 (refer to FIG. 22). FIG. 22 is an explanatory diagram illustratingan outline of an overlay display of the omnidirectional image and thetranslucent sound pressure heat map in the third exemplary embodiment.

In the exemplary embodiment, as illustrated in FIG. 22, monitoringapparatus 10 displays omnidirectional image IMG1 captured byomnidirectional camera CAk, on first monitor MN1. Monitoring apparatus10 generates the sound pressure heat map corresponding toomnidirectional image IMG1 by using the sound source visual image inwhich the sound pressure value calculated on a per-pixel basis usingpixels which form omnidirectional image IMG1 or on aper-predetermined-unit basis of the pixels is converted in stages into adifferent visual image, and further generates translucent sound pressureheat map TRL1 obtained by converting the sound pressure heat map intothe translucent image to display the result on second monitor MN2.

In FIG. 22, monitoring apparatus 10 respectively displaysomnidirectional image IMG1 and translucent sound pressure heat map TRL1on first monitor MN1 and second monitor MN2 which are separate monitors,but may display, for example, omnidirectional image IMG1 on a window andtranslucent sound pressure heat map TRL1 on a different window, inomnidirectional image IMG1.

Monitoring apparatus 10 displays omnidirectional image IMG1A, which isobtained by superimposing translucent sound pressure heat map TRL1 ontoomnidirectional image IMG1, on first monitor MN1 (refer to FIG. 23).FIG. 23 is a diagram illustrating an example of a display screen offirst monitor MN1 on which omnidirectional image IMG1 and translucentsound pressure heat map TRL1 are overlay-displayed.

In FIG. 23, in first monitor MN1, pilotless flying object do detected bymonitoring apparatus 10 is visible on the upper right side of the paperof FIG. 23 on omnidirectional image IMG1A. In addition, the translucentsound pressure heat map respectively corresponding to the sound sourcesgenerated by pilotless flying object dn or in the range of an imagingangle of view of omnidirectional camera CAk is superimposed anddisplayed.

As described with reference to FIG. 24, in the exemplary embodiment, ifthe sound pressure value on a per-pixel basis calculated by sound sourcedirection detector 34 is less than or equal to first threshold th1, thearea is displayed to be colorless translucent (that is, colorless), ifthe sound pressure value is greater than first threshold th1 and is lessthan or equal to second threshold th2, the area is displayed intranslucent blue, if the sound pressure value is greater than secondthreshold th2 and less than or equal to third threshold th3, the area isdisplayed in translucent pink, and if the sound pressure value isgreater than third threshold th3, the area is displayed in translucentred.

In FIG. 23, for example, in the vicinity of the rotary blade or therotor around the body center of pilotless flying object dn, since thesound pressure value is greater than third threshold th3, the areas arerendered by translucent red areas RD1A, RD2A, RD3A, and RD4A. Similarly,the areas around the translucent red areas are rendered by translucentpink areas PD1A, PD2A, PD3A, and PD4A which indicate that the soundpressure value is the greatest next to the translucent red area.Similarly, the areas around the translucent pink areas are rendered bytranslucent blue areas BD1A, BD2A, BD3A, and BD4A which indicate thatthe sound pressure value is the greatest next to the translucent pinkarea.

In addition, FIG. 23 illustrates that the sound source is also presentin office buildings, and pixels at which the sound pressure value on aper-pixel basis calculated by sound source direction detector 34 isgreater than third threshold th3 or sets of such pixels are rendered bytranslucent red areas R1A, R2A, R3A, R4A, R5A, R6A, and R7A. Similarly,the areas around the translucent red areas of the office buildings arerendered by translucent pink areas P1A, P2A, P3A, P4A, and P5A whichindicate that the sound pressure value is the greatest next to thetranslucent red area. Similarly, the areas around the translucent pinkareas of the office buildings are rendered by translucent blue areas B1Aand B2A which indicate that the sound pressure value is the greatestnext to the translucent pink area. Other areas in omnidirectional imageIMG1A are rendered by colorless areas since the sound pressure value isless than or equal to first threshold th1.

In this manner, in the exemplary embodiment, since monitoring apparatus10 superimposes translucent sound pressure heat map TRL1, which isdifferent from that of the first exemplary embodiment, ontoomnidirectional image IMG1 to display the result on first monitor MN1,it is possible for the user to visually determine the position of asound source appearing in omnidirectional image IMG1 and the volume of asound at the position, and further it is possible not to causedeterioration of visibility of omnidirectional image IMG1.

Next, an operation of pilotless flying object detection system 5 of theexemplary embodiment will be described with reference to FIG. 24.

FIG. 24 is a sequence diagram illustrating an example of an operationprocedure of the overlay display of omnidirectional image IMG1 andtranslucent sound pressure heat map TRL1 in the third exemplaryembodiment. When power is input to the devices (for example, firstmonitor MN1, monitoring apparatus 10, omnidirectional camera CAk, andmicrophone array MAk) of pilotless flying object detection system 5,pilotless flying object detection system 5 starts operating. Inaddition, in the description of FIG. 24, the masking area described inthe first exemplary embodiment may be used or may not be used. In FIG.24, a case in which the masking area is used is described as an example.In a case where the masking area is used, it is assumed that informationindicating the masking area is registered in memory 38.

Monitoring apparatus 10 performs an image transmission request inrelation to omnidirectional camera CAk (S71). Omnidirectional camera CAkstarts the imaging process corresponding to the input of power inaccordance with the image transmission request. In addition, monitoringapparatus 10 performs an audio transmission request in relation tomicrophone array MAk (S72). Microphone array MA starts the soundcollection process corresponding to the input of power in accordancewith the audio transmission request.

Once the initialization operations are completed, omnidirectional cameraCAk transmits the data of the omnidirectional image (for example, astill image or a video) which is obtained through imaging to monitoringapparatus 10 via network NW (S73). In order for the brief description,in FIG. 24, it is described that the omnidirectional image data istransmitted from omnidirectional camera CAk; however, two-dimensionalpanorama image data may be transmitted. Monitoring apparatus 10 convertsthe omnidirectional image data which is transmitted from omnidirectionalcamera CAk into display data such as NTSC, outputs the display data tofirst monitor MN1, and instructs first monitor MN1 to display theomnidirectional image data (S74). When the display data transmitted frommonitoring apparatus 10 is input, first monitor MN1 displays on thescreen, the data (refer to the upper left side of the paper of FIG. 22)of omnidirectional image IMG1 by omnidirectional camera CAk.

Microphone array MAk encodes the audio data of monitoring area 8 whichis obtained through sound collection and transmits the encoded audiodata to monitoring apparatus 10 via network NW (S75). In monitoringapparatus 10, sound source direction detector 34 calculates the soundpressure as the sound parameter, on a per-pixel basis using theindividual pixels which form the omnidirectional image data ofmonitoring area 8, based on the omnidirectional image data which iscaptured by omnidirectional camera CAk and the audio data which iscollected by microphone array MAk, and further estimates the soundsource position within monitoring area 8 (S76). When monitoringapparatus 10 detects pilotless flying object dn, the estimated soundsource position is used as the reference position of directivity settingarea BF1 which is necessary for the initial setting of the directivitysetting direction.

In addition, in monitoring apparatus 10, output controller 35 generatesa sound pressure map in which a calculated value of the sound pressureis allocated to the position of a pixel on a per-pixel basis using thepixels which form the omnidirectional image data, by using the soundpressure values on a per-pixel basis using the pixels which form theomnidirectional image data which are calculated by sound sourcedirection detector 34. Furthermore, output controller 35 generates atranslucent sound pressure heat map such as that illustrated in theupper right side of the paper of FIG. 22 by performing a colortransformation process on the sound pressure values on a per-pixel basisusing the pixels of the generated sound pressure map so as to obtain thevisual image (for example, colored image) such that it is easy for theuser to perform visual determination (S77). The method of generating thetranslucent sound pressure heat map is a procedure in which, forexample, output controller 35 primarily generates a sound pressure heatmap (refer to step S7 of FIG. 9), secondarily performs a process ofcausing the sound pressure heat map to be translucent, and therebygenerates the translucent sound pressure heat map.

Further, in monitoring apparatus 10, when signal processor 33 formssequential directionality for the area other than the masking area setby masking area setter 69 a, by using the audio data transmitted frommicrophone array MAk in step S75, detection determination of pilotlessflying object dn for each directivity setting direction in which thedirectionality is formed is performed (S78). The detection determinationprocess of pilotless flying object dn is described with reference toFIGS. 10 and 11, and thus will not be repeated.

In a case in which pilotless flying object dn is detected as a result ofthe detection determination process, output controller 35 in monitoringapparatus 10 instructs to superimpose the translucent sound pressureheat map generated in step S77, and the identification mark (notillustrated), which represents pilotless flying object dn which ispresent in the directivity setting direction detected in step S78, ontoomnidirectional image IMG1 which is displayed on the screen of firstmonitor MN1 and to display the result (S79).

First monitor MN1 combines (superimposes) the translucent sound pressureheat map on omnidirectional image IMG1 according to the instruction frommonitoring apparatus 10 and displays the result, and combines(superimposes) the identification mark (not illustrated) representingpilotless flying object dn on omnidirectional image IMG1 and displaysthe result (S80). Subsequently, the process of pilotless flying objectdetection system 5 returns to step S73, and processes of steps S73 toS80 are repeated until a predetermined event such as the power beingoperated to turn off, for example, is detected.

As described above, in pilotless flying object detection system 5 of theexemplary embodiment, monitoring apparatus 10 calculates the soundpressure specifying the volume of the sound of monitoring area 8 on aper-predetermined-unit basis of pixels, which form the captured image(omnidirectional image IMG1) of monitoring area 8, by using the audiodata collected by microphone array MAk. Monitoring apparatus 10generates a translucent sound pressure heat map in which the soundsource visual image, in which the sound pressure is converted into avisual image according to comparison between the calculated value of thesound pressure and a threshold relating to the volume of a sound, on aper-predetermined-unit basis of pixels, is linked to correspond to thevolume of the omnidirectional image of monitoring area 8. Monitoringapparatus 10 superimposes the translucent sound pressure heat map ontothe captured image of monitoring area 8 and displays the result on firstmonitor MN1.

In this manner, in pilotless flying object detection system 5, it isdifficult to visually present, to a user, the location in monitoringarea 8 of omnidirectional camera CAk where the pilotless flying objectis detected, and what kinds of sound source are present at whichlocations in monitoring area 8 without deterioration of the visibilityof the captured image of omnidirectional camera CAk.

In addition, a plurality of thresholds relating to the volume of a soundare provided, and thus monitoring apparatus 10 generates a translucentsound pressure heat map including plural kinds of sound source visualimages, by using sound source visual image in which the sound pressureis converted in stages into a different visual image, according tocomparison between the sound pressure and the plurality of thresholds,on a per-predetermined-unit basis of pixels. In this manner, inmonitoring apparatus 10, it is possible for the user to furtherexpressly determine the presence of the sound pressure having pluralkinds of levels prescribed by the plurality of thresholds, by the soundsource visual image among the omnidirectional image captured byomnidirectional camera CAk.

In the exemplary embodiment, monitoring apparatus 10 sets the maskingarea described in the first exemplary embodiment, and detects thepilotless flying object in an area other than the masking area by usingthe audio data collected by microphone array MAk and informationindicating the masking area. In a case where the pilotless flying objectis detected in an area other than the masking area, monitoring apparatus10 displays the sound source visual image, which indicates the volume ofthe sound generated by the pilotless flying object, on first monitor MN1in a translucent manner in the vicinity of the pilotless flying object(in other words, sound source position of the pilotless flying object)in the omnidirectional image. In this manner, since monitoring apparatus10 can exclude the masking area from the detection target of thepilotless flying object, it is possible to suppress deterioration of thedetection precision of the masking area and to improve the speed of thedetection process for the pilotless flying object. Monitoring apparatus10 displays the level of the volume of the sound output from thepilotless flying object, at the sound source position of pilotlessflying object do detected in an area other than the masking area, byusing the translucent image of the sound source visual image, andtherefore, it is possible not to cause deterioration of visibility ofthe captured image around the sound source position as well as thevolume of the sound.

In the exemplary embodiment, monitoring apparatus 10 changes the settingof a correspondence relationship between each threshold of a pluralityof thresholds defining the volume of a sound in stages and plural kindsof sound source visual images, according to the captured image of theimaging area. Monitoring apparatus 10 generates a translucent soundpressure heat map in which the sound source visual image on aper-predetermined-unit basis of pixels, is linked to correspond to thesize of the captured image of the imaging area, based on the calculatedvalue of the sound pressure and the changed correspondence relationship.In this manner, monitoring apparatus 10 can change the correspondencerelationship between the calculated value of the sound pressure obtainedon a per-pixel basis using pixels which form the captured image or on aper-predetermined-unit basis of the pixels, and the sound source visualimage corresponding to the calculated value of the sound pressure,according to the contents of the omnidirectional image (the capturedimage) captured by omnidirectional camera CAk. Accordingly, for example,at a location where a specific calculated value of the sound pressure isconcentrated, monitoring apparatus 10 uses not a sound source visualimage formed of a single color, but a sound source visual image formedof plural kinds of colors, for the sound source visual image around thelocation so as to cause the user to clearly ascertain in detail, thedistribution of the volume of the sound of the sound source appearing inthe captured image in detail.

Summary of Disclosure

Hereinafter, the summary of the disclosure will be described.

A monitoring system of the disclosure includes a camera which images animaging area; a microphone array which collects audio of the imagingarea; a monitor which displays a captured image of the imaging areawhich is captured by the camera; a masking area setter that sets amasking area to be excluded from detection of a pilotless flying objectwhich appears in the captured image of the imaging area, based on theaudio collected by the microphone array; a detector that detects thepilotless flying object based on the audio collected by the microphonearray and the masking area set by the masking area setter; and a signalprocessor that superimpose a sound source visual image, which indicatesthe volume of a sound at a sound source position, at the sound sourceposition of the pilotless flying object in the captured image anddisplays the result on the monitor in a case where the pilotless flyingobject is detected in an area other than the masking area.

The monitoring system according to the disclosure may further include asound parameter deriving unit that derives a sound parameter, whichspecifies the volume of a sound of the imaging area, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area, based on the audio collected by the microphone array,in which the masking area setter may superimpose and display a soundsource area in which the sound parameter derived by the sound parameterderiving unit is greater than or equal to a masking area thresholdrelating to the volume of a sound, on the monitor, and set the soundsource area displayed on the monitor as the masking area through auser's confirming operation.

In the monitoring system according to the disclosure, through a user'sadding operation for further adding a sound source area displayed on themonitor, the masking area setter may set the sound source area after theuser's adding operation as the masking area.

In the monitoring system according to the disclosure, through a user'sdeleting operation for deleting at least a part of a sound source areadisplayed on the monitor, the masking area setter may set the soundsource area after the user's deleting operation as the masking area.

In the monitoring system according to the disclosure, the signalprocessor superimposes the sound source visual image in which the soundparameter is converted in stages into a different visual image accordingto comparison between the derived sound parameter and a plurality ofthresholds relating to the volume of a sound, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area and displays the result on the monitor.

A monitoring system according to the disclosure may include a camerawhich images an imaging area; a microphone array which collects audio ofthe imaging area; a monitor which displays a captured image of theimaging area which is captured by the camera; a sound parameter derivingunit that derives a sound parameter, which specifies the volume of asound of the imaging area, on a per-predetermined-unit basis of pixelswhich form the captured image of the imaging area, based on the audiocollected by the microphone array; and a signal processor that generatesa sound parameter map as a translucent map in which a sound sourcevisual image, in which the sound parameter is converted into a visualimage according to comparison between the sound parameter derived by thesound parameter deriving unit and a threshold relating to the volume ofa sound, on a per-predetermined-unit basis of pixels, is linked tocorrespond to the size of the captured image of the imaging area, inwhich the signal processor may superimpose the translucent map onto thecaptured image of the imaging area and display the result on themonitor.

In the monitoring system according to the disclosure, a plurality ofthresholds relating to the volume of the sound may be provided, and thesignal processor may generate the sound parameter map as a translucentmap including plural kinds of sound source visual images, by using thesound source visual image in which the sound parameter is converted instages into a different visual image, according to comparison betweenthe sound parameter and the plurality of thresholds, on aper-predetermined-unit basis of pixels.

The monitoring system according to the disclosure may further include amasking area setter that sets a masking area to be excluded fromdetection of a pilotless flying object which appears in the capturedimage of the imaging area, based on the audio collected by themicrophone array; and a detector that detects the pilotless flyingobject based on the audio collected by the microphone array and themasking area set by the masking area setter, in which, in a case wherethe pilotless flying object is detected in an area other than themasking area, the signal processor may display the sound source visualimage, which indicates the volume of the sound of the pilotless flyingobject, on the monitor in a translucent manner in the vicinity of thepilotless flying object in the captured image of the imaging area.

The monitoring system according to the disclosure may further include athreshold adjuster that changes a setting of a correspondencerelationship between each threshold of a plurality of thresholdsdefining the volume of a sound in stages and the plural kinds of soundsource visual images according to the captured image of the imagingarea, and the signal processor may generate a sound parameter map as atranslucent map in which the sound source visual image on aper-predetermined-unit basis of pixels is linked to correspond to thesize of the captured image of the imaging area based on the soundparameter derived by the sound parameter deriving unit and thecorrespondence relationship changed by the threshold adjuster.

A monitoring system according to the discloser may include: a camerawhich images an imaging area; a microphone array which collects audio ofthe imaging area; a monitor which displays a captured image of theimaging area which is captured by the camera; a sound parameter derivingunit that derives a sound parameter, which specifies the volume of asound of the imaging area, on a per-predetermined-unit basis of pixelswhich form the captured image of the imaging area, based on the audiocollected by the microphone array; and a signal processor thatsuperimposes a sound source visual image in which the sound parameter isconverted in stages into a different visual image according tocomparison between the sound parameter derived by the sound parameterderiving unit and a plurality of thresholds relating to the volume of asound, on a per-predetermined-unit basis of pixels which form thecaptured image of the imaging area and displays the result on themonitor, in which, when any sound source position is designated in thecaptured image of the imaging area on which the sound source visualimage is superimposed, the sound parameter deriving unit may derive thesound parameter for each value obtained by dividing a predetermined unitof pixels which form a rectangular range including the sound sourceposition by a ratio between sizes of the captured image of the imagingarea and the rectangular range.

A monitoring system according to the disclosure may include: a camerawhich images an imaging area; a microphone array which collects audio ofthe imaging area; a monitor which displays a captured image of theimaging area which is captured by the camera; a sound parameter derivingunit that derives a sound parameter, which specifies the volume of asound of the imaging area, on a per-predetermined-unit basis of pixelswhich form the captured image of the imaging area, based on the audiocollected by the microphone array; a threshold adjuster that changes asetting of a correspondence relationship between each threshold of aplurality of thresholds defining the volume of a sound in stages and asound source visual image in which the sound parameter is converted instages into a different visual image according to comparison between thesound parameter and each threshold, according to the captured image ofthe imaging area; and a signal processor that superimposes the soundsource visual image corresponding to the sound parameter onto thecaptured image of the imaging area, on a per-predetermined-unit basis ofpixels which form the captured image of the imaging area, based on thesound parameter derived by the sound parameter deriving unit and thecorrespondence relationship changed by the threshold adjuster anddisplays the result.

In the monitoring system according to the disclosure, the thresholdadjuster may change widths of the thresholds defining the sound sourcevisual image, based on the frequency of appearance of the soundparameter on a per-predetermined-unit basis of pixels which form thecaptured image.

In the monitoring system according to the disclosure, the thresholdadjuster may equally change all other inter-threshold widths except forthe inter-threshold width that is changed, according to an operation ofchanging the inter-threshold width defining the use of the sound sourcevisual image corresponding to the upper limit value of the soundparameter.

In the monitoring system according to the disclosure, the thresholdadjuster may equally change all other inter-threshold widths except forthe inter-threshold width that is changed, according to an operation ofchanging the inter-threshold width defining the use of the sound sourcevisual image corresponding to the lower limit value of the soundparameter.

In the monitoring system according to the disclosure, the thresholdadjuster may equally change all other inter-threshold widths except forthe width of the upper end thresholds and the width of the lower endthresholds that are changed, according to an operation of changing theinter-threshold width defining the use of the sound source visual imagecorresponding to the upper limit value of the sound parameter and theinter-threshold width defining the use of the sound source visual imagecorresponding to the lower limit value of the sound parameter.

A monitoring method according to the disclosure, in a monitoring systemprovided with a camera and a microphone array, may include imaging animaging area by the camera; collecting audio of the imaging area by themicrophone array; displaying a captured image of the imaging area whichis captured by the camera, on a monitor; setting a masking area to beexcluded from detection of a pilotless flying object which appears inthe captured image of the imaging area, based on the audio collected bythe microphone array; detecting the pilotless flying object based on theaudio collected by the microphone array and the set masking area; andsuperimposing a sound source visual image, which indicates the volume ofa sound at a sound source position, at the sound source position of thepilotless flying object in the captured image and displaying the resulton the monitor in a case where the pilotless flying object is detectedin an area other than the masking area.

The monitoring method according to the disclosure, in a monitoringsystem provided with a camera and a microphone array, may includeimaging an imaging area by the camera; collecting audio of the imagingarea by the microphone array; displaying a captured image of the imagingarea which is captured by the camera, on a monitor; deriving a soundparameter, which specifies the volume of a sound of the imaging area, ona per-predetermined-unit basis of pixels which form the captured imageof the imaging area, based on the audio collected by the microphonearray; generating a sound parameter map as a translucent map in which asound source visual image, in which the sound parameter is convertedinto a visual image according to comparison between the derived soundparameter and a threshold relating to the volume of a sound, on aper-predetermined-unit basis of pixels, is linked to correspond to thesize of the captured image of the imaging area; and superimposing thegenerated translucent map onto the captured image of the imaging areaand displaying the result on the monitor.

A monitoring method according to the disclosure, in a monitoring systemprovided with a camera and a microphone array, may include imaging animaging area by the camera; collecting audio of the imaging area by themicrophone array; displaying a captured image of the imaging area whichis captured by the camera, on a monitor; deriving a sound parameter,which specifies the volume of a sound of the imaging area, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area, based on the audio collected by the microphone array;superimposing a sound source visual image in which the sound parameteris converted in stages into a different visual image according tocomparison between the sound parameter and a plurality of thresholds, ona per-predetermined-unit basis of pixels which form the captured imageof the imaging area and displaying the result on the monitor; andfurther deriving, when any sound source position is designated in thecaptured image of the imaging area on which the sound source visualimage is superimposed, the sound parameter for each value obtained bydividing a predetermined unit of pixels which form a rectangular rangeincluding the sound source position by a ratio between sizes of thecaptured image of the imaging area and the rectangular range.

A monitoring method according to the disclosure, in a monitoring systemprovided with a camera and a microphone array, may include imaging animaging area by the camera; collecting audio of the imaging area by themicrophone array; displaying a captured image of the imaging area whichis captured by the camera, on a monitor; deriving a sound parameter,which specifies the volume of a sound of the imaging area, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area, based on the audio collected by the microphone array;changing a setting of a correspondence relationship between eachthreshold of a plurality of thresholds defining the volume of a sound instages and a sound source visual image in which the sound parameter isconverted in stages into a different visual image according tocomparison between the sound parameter and each threshold, according tothe captured image of the imaging area; and superimposing the soundsource visual image corresponding to the sound parameter onto thecaptured image of the imaging area, on a per-predetermined-unit basis ofpixels which form the captured image of the imaging area, based on thederived sound parameter and the changed correspondence relationship anddisplaying the result on the monitor.

Hereunto description is given of an exemplary embodiment with referenceto the drawings, and it goes without saying that the disclosure is notlimited to the examples given. It is clear to a person skilled in theart that various modifications and corrections may be made within thescope disclosed in the claims. Naturally, such modifications andcorrections are understood to fall within the technical scope of thedisclosure.

What is claimed is:
 1. A monitoring system comprising: a camera whichimages an imaging area; a microphone array which collects audio of theimaging area; a monitor which displays a captured image of the imagingarea which is captured by the camera; a processor; and a memoryincluding instructions that, when executed by the processor, cause theprocessor to perform operations including: setting a masking area to beexcluded from detection of a pilotless flying object which appears inthe captured image of the imaging area, based on the audio collected bythe microphone array; detecting the pilotless flying object based on theaudio collected by the microphone array and the masking area set by themasking area setter; and superimposing a sound source visual image,which indicates the volume of a sound at a sound source position, at thesound source position of the pilotless flying object in the capturedimage and displays the result on the monitor in a case where thepilotless flying object is detected in an area other than the maskingarea.
 2. The monitoring system of claim 1, the operations furtherincluding: deriving a sound parameter, which specifies the volume of asound of the imaging area, on a per-predetermined-unit basis of pixelswhich form the captured image of the imaging area, based on the audiocollected by the microphone array, superimposing and displaying a soundsource area in which the sound parameter derived by the sound parameterderiving unit is greater than or equal to a masking area thresholdrelating to the volume of a sound, on the monitor, and setting the soundsource area displayed on the monitor as the masking area through auser's confirming operation.
 3. The monitoring system of claim 2,wherein through a user's adding operation for further adding a soundsource area displayed on the monitor, the operations further including:setting the sound source area after the user's adding operation as themasking area.
 4. The monitoring system of claim 2, wherein through auser's deleting operation for deleting at least a part of a sound sourcearea displayed on the monitor, the operations further including: settingthe sound source area after the user's deleting operation as the maskingarea.
 5. The monitoring system of claim 2, the operations furtherincluding: superimposing the sound source visual image in which thesound parameter is converted in stages into a different visual imageaccording to comparison between the derived sound parameter and aplurality of thresholds relating to the volume of a sound, on aper-predetermined-unit basis of pixels which form the captured image ofthe imaging area and displays the result on the monitor.
 6. A monitoringmethod in a monitoring system provided with a camera and a microphonearray, the method comprising: imaging an imaging area by the camera;collecting audio of the imaging area by the microphone array; displayinga captured image of the imaging area which is captured by the camera, ona monitor; setting a masking area to be excluded from detection of apilotless flying object which appears in the captured image of theimaging area, based on the audio collected by the microphone array;detecting the pilotless flying object based on the audio collected bythe microphone array and the set masking area; and superimposing a soundsource visual image, which indicates the volume of a sound at a soundsource position, at the sound source position of the pilotless flyingobject in the captured image and displaying the result on the monitor ina case where the pilotless flying object is detected in an area otherthan the masking area.